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DESCRIPTION 

IMAGE SIMILARITY CALCULATION SYSTEM, IMAGE SEARCH SYSTEM, 

IMAGE SIMILARITY CALCULATION METHOD, AND 
IMAGE SIMILARITY CALCULATION PROGRAM 



Technical Field 
[0001] 

The present invention relates to an image similarity 
calculation system, an image similarity calculation method, an 

10 image similarity calculation program for calculating 

similarities of multiple images, and an image retrieval system 
for retrieving an image similar to an inquiry image. More 
specifically, the present invention relates to an image 
similarity calculation system, an image similarity calculation 

15 method, an image similarity calculation program for calculating 
a similarity between an original image and an edited image, and 
an image retrieval system for retrieving an image similar to an 
edited inquiry image . 



20 Background Art 
[0002] 

There is an image similarity calculation method of 
calculating similarity between two images to be compared with each 
other. Generally, such image similarity calculation method is 
25 used for calculating a similarity between an inquiry image and 
an image for retrieval , in an image retrieval system for 
retrieving an image similar to the inquiry image from a group of 
registered images . 
[0003] 

30 Patent Document 1 describes an example of the conventional 

image retrieval method. The image retrieval method described in 
Patent Document 1 divides an image into multiple blocks and 
calculates the feature quantity of each block , Further , the image 
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retrieval method provides each block with a label in accordance 
with the feature quantity obtained for each block and generates 
a label matrix by arranging labels in a specified order. The image 
retrieval method calculates an image similarity between the 
5 inquiry image and an image for retrieval as a similarity of label 
columns obtained from these images. In retrieval, images for 
retrieval for which similarities exceed a threshold value are 
output as a retrieval result. 
[0004] 

10 Patent Document 2 describes an image retrieval method of 

retrieving an image partially similar to an inquiry image. The 
image retrieval method described in Patent Document 2 inputs an 
inquiry image and an inquiry area, i.e., an area for retrieval 
in the inquiry image. The method divides an inquiry image into 

15 blocks. A weight is defined in accordance with an overlapping 
area between each block and the inquiry area. The method uses 
the weight to weight the feature quantity of each block to 
calculate the feature quantity. Similarly, the method 
calculates the feature quantity of an image for retrieval and 

20 compares both feature quantities to calculate the image 

similarity. In retrieval, images with high image similarities 

are output as a retrieval result. 

[0005] 

Patent Document 3 describes an image retrieval method 
25 reflecting user*s retrieval intention. The image retrieval 

method described in Patent Document 3 divides an inquiry image 
and an image for retrieval into multiple areas and allows a user 
to define different weights for the divided areas. The method 
obtains a similarity by comparing feature quantities of the 
30 corresponding divided areas between the inquiry image and the 
image for retrieval. The method applies the weight defined for 
each divided area to the similarity to calculate an image 
similarity. Since a user can specify an area that needs special 
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attention during retrieval, it is possible to perform the image 

retrieval reflecting the user's retrieval intention. 

[0006] 

Patent Document 1: JP-A-1999-312248 (paragraphs 0012 
5 through 0013, FIGS. 9 and 13) 

Patent Document 2: JP-A-2000-163576 (paragraphs 0017 
through 0020, FIGS. 7 and 8) 

Patent Document 3: JP-A-2001-134765 (paragraphs 0028 
through 0031, FIG. 11) 

10 

Disclosure of the Invention 
[0007] 

However, the method described in Patent Document 1 assumes 
identity between an inquiry image and an image for retrieval . An 

15 image may be variously edited by superposing a ticlcer on it or 
partially cutting or cropping the image to partially change the 
image's pattern. The method calculates a small value for the 
image similarity between such changed image and an original image 
before the editing. There is a problem of inability to obtain 

20 sufficient throughput to determine or retrieve an original image 
from the edited image. 
[0008] 

While the method described in Patent Document 2 can deal 
with an edited image with partially different patterns, it is 

25 necessary to previously specify an area targeted for the retrieval . 
Lilcewise, the method described in Patent Document 3 assigns a 
weight value to each area in the image, necessitating a user to 
specify an area that needs special attention during retrieval. 
Accordingly, the user needs to previously specify an unedited area 

30 to calculate the image similarity for the purpose of determining 
or retrieving an original image before the editing from the edited 
image. There is a problem of causing great inconvenience to 
users . 
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[0009] 

The present invention has been made to solve the 
above-mentioned problems. An object of the present invention is 
therefore to provide an image similarity calculation system, an 
5 image retrieval system, an image similarity calculation method, 
and an image similarity calculation program capable of 
calculating the image similarity as a large value between an 
edited image and an original image before editing without causing 
inconvenience of specifying an area to users so as to accurately 
10 determine or retrieve the original image before editing from the 
edited image with partially changed patterns . 
[0010] 

According to the present invention, there is provided an 
image similarity calculation system, an image similarity 

15 calculation method, and an image similarity calculation program 
configured to: use a probability model of a probability for an 
editing process to be applied to an image; compare a feature 
quantity for each divided small region of the inquiry image with 
a feature quantity for each divided small region of the reference 

20 image; and calculate an image similarity between an inquiry image 
and a reference image. 
[0011] 

According to another embodiment of the present invention, 
there is provided an image similarity calculation system, an image 

25 similarity calculation method, and an image similarity 

calculation program configured to: compare a feature quantity for 
each divided small region of an inquiry image with a feature 
quantity for each divided small region of an reference image, 
taking into account a probability model of a probability for an 

30 editing process to be applied to each region of an image; and 
calculate an image similarity between the inquiry image and the 
reference image, 
[0012] 
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According to still another embodiment of the present 
invention, there is provided an image similarity calculation 
system, an image similarity calculation method, and an image 
similarity calculation program configured to: compare a feature 
5 quantity for each divided small region of an inquiry image with 
a feature quantity for each divided small region of a reference 
image; and calculate an image similarity between the inquiry image 
and the reference image, using a probability model of a 
probability for an editing process to be applied to an image. 
10 [0013] 

According to yet another embodiment of the present 
invention, an image similarity calculation system, an image 
similarity calculation method, and an image similarity 
calculation program configured to: compare a feature quantity for 

15 each divided small region of an inquiry image with a feature 
quantity for each divided small region of a reference image, using 
a probability model of a probability for an editing process to 
be applied to each region of an image; and calculate image 
similarity between the inquiry image and the reference image, 

20 using the probability model. 
[0014] 

According to a preferred embodiment of the present 
invention, for example, there is provided an image similarity 
calculation system comprising: local region weight calculation 

25 means configured to calculate an image-region-based weight value 
based on a probability for an editing process to be applied to 
an image for each region; small region similarity calculation 
means configured to compare a feature quantity for each divided 
small region of an inquiry image with a feature quantity for each 

30 divided small region of a reference image to calculate a 
similarity of feature quantities for the respective small 
regions; and image similarity calculation means configured to 
apply a region-based weight value to the calculated 
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small-region-based similarity to calculate an image similarity 
between the inquiry image and the reference image. The 
above-mentioned construction is used to calculate the image 
similarity in consideration for editing probability for each 
5 image region. This makes it possible to eliminate the effect of 
a highly possibly edited region during determination of image 
similarities . It is possible to yield a large value for the image 
similarity between the edited image and the original image. 
[0015] 

10 According to a preferred embodiment of the present 

invention, for example, there is provided an image similarity 
calculation system comprising: match determination threshold 
determination means configured to calculate a threshold value of 
determining a match between images based on a probability for an 

15 editing process to be applied to an image for each region; small 
region similarity calculation means configured to compare a 
feature quantity for each small region resulting from dividing 
an inquiry image with a feature quantity for each small region 
resulting from dividing a reference image to calculate a 

20 similarity of feature quantities for the respective small 

regions; and image similarity calculation means configured to 
calculate an overall image's similarity from the calculated 
small-region-based similarities, modify the overall image's 
similarity using a threshold value to determine a match, and 

25 calculate an image similarity between the inquiry image and the 
reference image. By this construction, probability distribution 
of similarities between images is estimated based on the 
probability model of a probability for an editing process to be 
applied to an image, and an optimum threshold value to determine 

30 whether or not there is a match between two images to be compared 
is calculated. This threshold value is used to modify the 
similarity between images. It is possible to yield a large value 
for the image similarity between the edited. image and the original 
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image and to yield a small value for the image similarity between 
two images completely irrelevant to each other. Further, it is 
possible to calculate an image similarity suited for determining 
whether or not an edited inquiry image is generated from the 
5 reference image. 
[0016] 

According to a preferred embodiment of the present 
invention, for example, there is provided an image retrieval 
system comprising: editing probability model estimation means 

10 configured to calculate local region editing probability, i.e. , 
probability for an editing process to be applied an image for each 
local region using a learning image or a device characteristic 
supplied as an input; local region weight calculation means 
configured to calculate a local region weight value, i.e., a 

15 weight value for each local region in an image based on the local 
region editing probability; image division means configured to 
divide an edited inquiry image supplied as input into small 
regions; small region feature quantity extraction means 
configured to extract an inquiry image small region feature 

20 quantity, i.e. , a feature quantity for each small region from the 
divided small regions; reference image group small region feature 
quantity storage means configured to store small region feature 
quantities for respective reference images in a reference image 
group composed of a plurality of previously registered reference 

25 images; small region similarity calculation means configured to 
compare a small region feature quantity of each reference image 
stored in the reference image group small region feature quantity 
storage portion with the inquiry image small region feature 
quantity and to calculate a small region similarity, i.e., a 

30 similarity of small-region-based feature quantities for each 
reference image; image similarity calculation means configured 
to weight a small region similarity calculated by the small region 
similarity calculation means for each of the reference images 
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using a small-region-based weight value found from the local 
region weight value, and to calculate an image similarity between 
the inquiry image and each reference image in the reference image 
group; and retrieval result output means configured to output a 
5 retrieval result for the inquiry image from the reference image 
group based on an image similarity calculated by the image 
similarity calculation means for each reference image. The 
above-mentioned construction is used to calculate an image 
similarity in consideration for image-region-based editing 
10 probability. This makes it possible to eliminate the effect of 
a highly possibly edited region during determination of image 
similarities. It is possible to yield a large value for the image 
similarity between the edited image and the original image. 
[0017] 

15 A first effect of the present invention is to be capable 

of yielding a large value for the image similarity between an 
edited image and the original image. The reason is that the 
present invention calculates an image similarity in consideration 
for a probability model of a probability for an editing process 

20 to be applied to images and is therefore capable of eliminating 
the effect of a highly possibly edited region during calculation 
of image similarities. 
[0018] 

A second effect of the present invention is to be capable 
25 of accurately retrieving the original image from an edited image. 
The reason is that the present invention retrieves images using 
the image similarity calculation method having the first effect 
capable of yielding a large value for the image similarity between 
an edited image and the original image. 

30 

Brief Description of the Drawings 
[0019] 

FIG. 1 is a block diagram showing the configuration of a 
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first embodiment of the present invention; 

FIG. 2 is a flowchart showing operations of the first 
embodiment of the present invention; 

FIG. 3 is an explanatory diagram showing how to calculate 
local region editing probability; 

FIG. 4 is a block diagram showing a modification of the first 
embodiment ; 

FIG. 5 is an explanatory diagram showing an example of local 
region editing probability stored in editing probability model 
storage means ; 

FIG. 6 is a block diagram showing another modification of 
the first embodiment; 

FIG. 7 is a block diagram showing the configuration of a 
second embodiment of the present invention; 

FIG. 8 is a flowchart showing operations of the second 
embodiment of the present invention; 

FIG. 9 is a block diagram showing the configuration of a 
third embodiment of the present invention; 

FIG. 10 is a block diagram showing the configuration of a 
fourth embodiment of the present invention; 

FIG. 11 is a block diagram showing the configuration of a 
fifth embodiment of the present invention; 

FIG. 12 is a flowchart showing operations of the fifth 
embodiment of the present invention; 

FIG. 13 is a block diagram showing the configuration of a 
sixth embodiment of the present invention; 

FIG. 14 is a flowchart showing operations of the sixth 
embodiment of the present invention; 

FIG. 15 is a block diagram showing the configuration of a 
seventh embodiment of the present invention; 

FIG. 16 is a block diagram showing the configuration of an 
eighth embodiment of the present invention; 

FIG. 17 is a block diagram showing the configuration of a 
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ninth embodiment of the present invention; 

FIG. 18 is a block diagram showing the configuration of a 
tenth embodiment of the present invention; 

FIG. 19 is a block diagram showing the configuration of. an 
eleventh embodiment of the present invention; and 

FIG. 20 is a block diagram showing the configuration of a 
twelfth embodiment of the present invention. 

Explanation of Reference Numerals 

[0020] 

11: Feature quantity extraction portion 
111: Image division means 

112: Small region feature quantity extraction means 

12: Image similarity calculation portion 

121: Small region similarity calculation means 

122: Image similarity calculation means 

13 : Editing probability model estimation means 

139: Editing probability model storage means 

14: Local region weight calculation means 

22 : Image similarity calculation portion 

221: Small region similarity calculation means 

222 : Image similarity calculation means 

25: Reference image group small region feature quantity storage 
portion 

26: Retrieval result output means 

32: Image similarity calculation portion 

321: Small region similarity calculation means 

322: Image similarity calculation means 

33: Editing region detection means 

34: Local region weight calculation means 

42: Image similarity calculation portion 

421: Small region similarity calculation means 

422: Image similarity calculation means 
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52: Image similarity calculation portion 

521: Small region similarity calculation means 

522: Image similarity calculation means 



53 
54 
62 



Editing probability model estimation means 
Match determination threshold calculation means 
Image similarity calculation portion 



621: Small region similarity calculation means 
622: Image similarity calculation means 
72 : Image similarity calculation portion 
721: Small region similarity calculation means 
722: Image similarity calculation means 



73 
74 
82 
821: 
822: 
92: 
921 : 
922 : 



20 93 
94 
95 



1020 
1021 
25 1022 
1120 
1121 
1122 
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114 

115 



Editing region detection means 

Match determination threshold calculation means 
Image similarity calculation portion 
Small region similarity calculation means 
Image similarity calculation means 
Image similarity calculation portion 
Small region similarity calculation means 
Image similarity calculation means 
Editing probability model estimation means 
Local region weight calculation means 
Match determination threshold calculation means 
Image similarity calculation portion 
Small region similarity calculation means 
Image similarity calculation means 
Image similarity calculation portion 
Small region similarity calculation means 
Image similarity calculation means 
Editing region detection means 
Local region weight calculation means 
Match determination threshold calculation means 



1220: Image similarity calculation portion 
1221: Small region similarity calculation means 
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1222: Image similarity calculation means 

Best Mode for Carrying Out the Invention 
[0021] 

5 The best mode for carrying out the present invention will 

be described in detail with reference to the accompanying 
drawings . 
[0022] 

[First Embodiment] 

10 A first embodiment of the present invention will be 

described with reference to the accompanying drawings. As shown 
in a block diagram of FIG. 1, the first embodiment of the image 
similarity calculation system according to the present invention 
is functionally composed of a feature quantity extraction portion 

15 11, an image similarity calculation portion 12, an editing 
probability model estimation means (an example of editing 
probability calculation means) 13, and a local region weight 
calculation means 14. 
[0023] 

20 The feature quantity extraction portion 11 functionally 

includes an image division means 111 and a small region feature 
quantity extraction means 112. 
[0024] 

When an edited inquiry image is input, the image division 
25 means 111 divides the inquiry image into small regions. The image 
division means 111 outputs the inquiry image divided into small 
regions to the small region feature quantity extraction means 112 . 
The editing process signifies partially changing an image pattern. 
Editing examples include overlaying superposing objects such as 
30 a ticker, a caption, and a character on an image, partially cutting 
or cropping an image, and partially mosaicking or blurring an 
image. There is available any method of dividing an image into 
small regions . The method of dividing an image into small regions 
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provides examples such as dividing an image into equally sized 
rectangles, dividing an image into irregularly sized rectangles, 
and dividing an image into diversely sized shapes such as circles 
and triangles other than rectangles. However, the present 
invention is not limited to these division methods. An image may 
be divided so that small regions partially overlap with each other. 
The entire image need not be divided into small regions. Only 
part of the image may be divided into small regions . 
[0025] 

The small region feature quantity extraction means 112 
extracts the feature quantity for each small region from an 
inquiry image that is divided into small regions input from the 
image division means 11. The small region feature quantity 
extraction means 112 outputs the extracted small region feature 
quantity for the inquiry image to the small region similarity 
calculation means 121. Examples of small region feature 
quantities to be extracted include color information, edge 
information, texture information, shape information, and motion 
information. The present invention is not limited thereto. 
[0026] 

The small region feature quantities concerning the color 
information are exemplified by representative values such as an 
average value , a mode value , and a median value for all pixel values 
in each small region, and an occurrence histogram for pixel values . 
Pixel values signify coordinate values used for various color 
spaces such as RGB color space, HSV color space, YUV color space, 
YIQ color space, YCbCr color space, L*a*b* color space, and XYZ 
color space. It may be preferable to use feature quantities 
concerning color information such as Dominant Color, Color Layout, 
Scalable Color, and Color Structure specified in international 
standard ISO/IEC 15938-3. 
[0027] 

As an example of the small region feature quantity 



concerning the edge information, it may be preferable to use a 
feature quantity (hereafter referred to as an edge element feature 
quantity) to describe the global edge pattern for each small 
region as the best matching pattern among the multiple edge 
5 patterns. Further, it may be preferable to use an occurrence 
histogram for a local edge pattern in each small region. Moreover, 
it may be preferable to use Edge Histogram specified in ISO/IEC 
15938-3. Examples of feature quantities concerning the texture 
information include Homogeneous Texture and Texture Browsing 
10 specified in ISO/1EC15938-3 . Examples of feature quantities 
concerning the shape information include Region Shape, Contour 
Shape, and Shape 3D specified in ISO/IEC 15938-3. 
[0028] 

When the inquiry image is a frame image in a moving picture, 
15 it may be preferable to use the motion information as the small 
region feature quantity. Examples of feature quantities for the 
motion information include Parametric Motion and Motion Activity 
specified in ISO/IEC 15938-3. 
[0029] 

20 The editing probability model estimation means 13 

calculates local region editing probability, i.e., probability 
of an editing process applied to each local region in an image, 
from an edited learning image (/s) or characteristics of a device 
to perform the editing process supplied as input. The editing 

25 probability model estimation means 13 outputs the calculated 
local region editing probability to the local region weight 
calculation means 14, It is preferable that the local region 
corresponds to a small region divided by the image division means 
111. In addition, the local region may correspond to any of 

30 multiple smaller regions resulting from further dividing a small 
region divided by the image division means 111. Moreover, the 
local region may correspond to each element of the image. 
Furthermore, the local region may correspond to any of regions 
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divided in any shapes and sizes different from those for the small 
region division method for the image division means 111. 
[0030] 

The learning image or images signify one or more edited 
5 images used as a basis for the editing probability model 
estimation means 13 to calculate the local region editing 
probability. When the present invention is applied to a video 
editing apparatus, for example, a frame image constituting any 
edited video can be used as the learning image . The learning image 

10 is not limited to a single type. The learning image may be 

categorized into types characteristic to places on a screen for 
applying editing processes such as superposing tickers. 
Examples of the types include news images used as a news program, 
sports images used as a sports program, and variety images used 

15 as a variety program. For example, a sports image is assumed to 
have high possibility of applying an editing process to 
peripheries of the screen. A variety program image is assumed 
to have high possibility of applying an editing process to the 
center of the screen. In this manner, it is a good practice to 

20 use the learning image for sports images independently of the 
learning image for variety program images. In this case, the 
editing probability model estimation means 13 calculates the 
type-based local region editing probability. A user can specify 
a type of inquiry image to use the local region editing probability 

25 corresponding to the type of inquiry image. 
[0031] 

When an edited learning image (/s) is input, the local region 
editing probability is measured by specifying regions that an 
editing process was applied in the learning image. Methods of 
30 specifying edited regions may include not only manual methods, 
but also methods of automatically detecting editing processes 
such as the method (hereafter referred to as a ticker detection 
. method) of separating image information from text information as 



15 



described in JP-A-2003-179888 . Characteristics of a device to 
perform editing processes may be supplied as input. In such case, 
there is available the method of measuring the local region 
editing probability based on editing patterns applied to an image 
by the device and based on information about the frequency of using 
each editing patterns . 
[0032] 

Examples of the device characteristics include editing 
patterns applied by an editing device to an image and information 
about the frequency of using each editing patterns. For example, 
the present invention may be applied to video editing apparatuses . 
Using a given video editing apparatus may result in high 
possibility of processing one or more specific regions on a screen. 
Using another video editing apparatus may result in high 
possibility of editing one or more different specific regions on 
the screen. In this case, it is possible to use information 
indicating these regions to be edited as information indicative 
of editing patterns. Furthermore, for example, a sports image 
is assumed to have high possibility of applying an editing process 
to peripheral regions of the screen. A variety program image is 
assumed to have high possibility of applying an editing process 
to the center region of the screen. In this case, it is possible 
to use information indicating these regions to be edited as 
information indicative of editing patterns. That is, the device 
characteristics include not only the device characteristics as 
originally implied, but also editing patterns and the information 
about the frequency of using each editing patterns when a device 
(e.g., a ticker insertion apparatus) is used to vary editing 
patterns depending on image types . 
[0033] 

The local region weight calculation means 14 calculates a 
local region weight value based on the local region editing 
probability supplied from the editing probability model 



estimation means 13. The local region weight value quantifies 
how effectively the feature quantity extracted from local regions 
of an image works for calculation of the similarity for the overall 
image. The local region weight calculation means 14 outputs the 
5 calculated local region weight value to the image similarity 
calculation means 122. It is effective to set a larger local 
region weight value in proportion to a decrease in the local region 
editing probability attributed to the local region. There is an 
example of the method for calculating the local region weight 
10 value as follows. 

Local region weight value = 1 - local region editing 
probability 

The present invention is not limited thereto . A small local 
region weight value may be assigned when the local region editing 

15 probability is high. A large local region weight value may be 
assigned when the local region editing probability is low . During 
the image similarity calculation process, it may be preferable 
to use the other calculation methods capable of eliminating the 
effect of a highly possibly edited region. 

20 [0034] 

Like the local region in the editing probability model 
estimation means 13, it is preferable that the local region 
corresponds to a small region divided by the image division means 
111. In addition, the local region may correspond to any of 

25 multiple smaller regions resulting from further dividing a small 
region divided by the image division means 111 . Furthermore, the 
local region may correspond to each element of the image. Still 
further, the local region may correspond to any of regions divided 
in any shapes and sizes different from those for the small region 

30 division method for the image division means 111. 
[0035] 

The image similarity calculation portion 12 functionally 
includes the small region similarity calculation means 121 and 
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the image similarity calculation means 122 . 
[0036] 

The small region similarity calculation means 121 compares 
the small region feature quantity for an inquiry image supplied 
5 from the small region feature quantity extraction means 112 with 
the small region feature quantity for a reference image supplied 
as input. The small region similarity calculation means 121 
calculates a small region similarity, i.e., a similarity of the 
feature quantity for each small region. The small region 

10 similarity calculation means 121 outputs the calculated small 
region similarity to the image similarity calculation means 122. 
Any method of calculating the small region similarity can be 
settled according to the small region feature quantity to be used. 
For example, when the small region feature quantity to be used 

15 is a representative value such as an average value, a mode value, 
or a median value for the small region's color, there is a method 
of calculating the similarity based on a distance value and a 
correlation value in a color space being used. When the small 
region feature quantity to be used is an occurrence histogram for 

20 colors and edge elements, there is a method of calculating the 
similarity between histograms. 
[0037] 

The small region feature quantity to be used may be Dominant 
Color, Color Layout, Scalable Color, Color Structure, Edge 

25 Histogram, Homogeneous Texture, Texture Browsing, Region Shape, 
Contour Shape, Shape 3D, Parametric Motion, or Motion Activity 
specified in ISO/IEC 15938-3. In this case, similarities can be 
calculated in accordance with the distance (similarity) 
calculation method recommended by ISO/IEC 15938-3. 

30 [0038] 

The similarity may be a binary determination result such 
as 1 or 0. When the edge element feature quantity is used as the 
small region feature quantity, for example, there is a method of 
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setting the similarity to 1 if the comparing edge elements match 
or setting the similarity to 0 if the comparing edge elements do 
not match. The reference image supplied as input contains a small 
region concerning the small region feature quantity. It is 
5 preferable that this small region corresponds to a small region 
concerning the small region feature quantity for the inquiry image 
input from the small region feature quantity extraction means 112. 
This is because corresponding small regions can be directly 
compared with each other. 
10 [0039] 

Different image division methods may cause a discrepancy 
between the inquiry image • s small region and the reference image ' s 
small region. In this case, it is a good practice to use either 
image's small region feature quantity to complement the feature 

15 quantity corresponding to the other image's small region and then 
calculate the similarity. For example, a small region concerning 
the reference image's small region feature quantity may result 
from further dividing the small region concerning the inquiry 
image's small region feature quantity into multiple smaller 

20 regions. In this case, it is possible to generate the feature 
quantity corresponding to the inquiry image's small region by 
calculating an average value for the reference image's multiple 
small region feature quantities. There may be a case where the 
small region division method concerning the reference image's 

25 small region feature quantity uses shapes and sizes completely 
different from those for the small region division method 
concerning the inquiry image* s small region feature quantity. In 
this case, small region feature quantities of the reference 
image * s small regions which overlap with the inquiry image ' s small 

30 regions are then weighted in proportion to overlapping areas and 
are totaled. In this manner, it is possible to generate the 
feature quantity corresponding to the inquiry image's small 
region . 



19 



•A 



[0040] 

The small region similarity calculation means 121 inputs 
the small region similarity to the image similarity calculation 
means 122. The image similarity calculation means 122 weights 
5 this small region similarity using a weight value for each small 
region found from a local region weight value input from the local 
region weight calculation means 14. The image similarity 
calculation means 122 calculates an image similarity, i.e., a 
similarity for the overall image, from the resulting weighted 
10 small region similarities and outputs the calculated image 

similarity. Methods of calculating the image similarity include 
finding a total sum of weighted small region similarities for all 
small regions or finding an average value thereof. 
[0041] 

15 If the local region of the local region weight value input 

from the local region weight calculation means 14 corresponds to 
a small region of the small region similarity input from the small 
region similarity calculation means 121, the local region weight 
value can be directly used as a weight value for each small region. 

20 When the small region of the small region similarity does not 
correspond to the local region of the local region weight value, 
it is a good practice to use the local region weight value to 
complement the weight value corresponding to the small region of 
the small region similarity. For example, a local region of the 

25 local region weight value may result from further dividing a small 
region of the small region similarity into multiple smaller 
regions. In this case, it is possible to generate a weight value 
corresponding to the small region of the small region similarity 
by calculating an average value for multiple local region weight 

30 values . There may be a case where the local region division method 
concerning local region weight values uses shapes and sizes 
completely different from those for the small region division 
method concerning small region similarities. In this case. 
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weight values of the multiple local regions which overlap with 
the small regions are then weighted in proportion to overlapping 
areas and are totaled. In this manner, it is possible to generate 
the weight value corresponding to the small region of the small 
5 region similarity. 
[0042] 

The image similarity calculation system is implementable 
on a computer. The image similarity calculation system is 
composed of such components as the feature quantity extraction 

10 portion 11, the image similarity calculation portion 12, the 
editing probability model estimation means 13, and the local 
region weight calculation means 14. These components can be 
implemented as programs that allow the computer's central 
processing unit (CPU) to perform the above-mentioned functions. 

15 That is, the components constituting the image similarity 

calculation system are implementable on a computer and can be 
implemented as programs. This applies not only to the first 
embodiment, but also to the other embodiments to be described 
later . 

20 [0043] 

Operations of the first embodiment will now be described 
with reference to the block diagram of FIG. 1 and a flowchart of 
FIG. 2. 
[0044] 

25 A learning image or a device characteristic is supplied as 

input (step Al) . The editing probability model estimation means 
13 calculates a local region editing probability from the input 
learning image or device characteristic, i.e., a probability of 
applying the editing process to each local region of the image 

30 (step A2) . 
[0045] 

FIG. 3 is an explanatory diagram showing how the editing 
probability model estimation means 13 calculates local region 
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editing probability. According to the example shown in FIG. 3^ 
an image region corresponding to image data is divided into 16 
local regions. The editing probability model estimation means 
13 indicates "1" when the editing process is applied to respective 
5 local regions for each image in multiple learning images . 

Otherwise, the editing probability model estimation means 13 
indicates "0". The editing probability model estimation means 
13 performs a process such as averaging each local region to 
calculate the probability that the editing process is applied to 

10 each local region. That is, the editing probability model 

estimation means 13 estimates a probability model indicative of 
local region editing probabilities. Here is described a mere 
example of the method of estimating a probability model. The 
other methods may be used. 

15 [0046] 

The editing probability model estimation means 13 outputs 
the calculated local region editing probability to the local 
region weight calculation means 14. Based on the local region 
editing probability, the local region weight calculation means 
20 14 calculates a local region weight value, i.e. , a weight value 
for each local region in the image (step A3) . 
[0047] 

An edited inquiry image is supplied as input (step A4) . The 
image division means 111 divides the input inquiry image into 

25 small regions and outputs the image divided into small regions 
to the small region feature quantity extraction means 112 (step 
A5) . The small region feature quantity extraction means 112 
extracts an inquiry image small region feature quantity, i.e., 
a feature quantity for each of small regions of the divided image. 

30 The small region feature quantity extraction means 112 then 

outputs the extracted inquiry image small region feature quantity 
to the small region similarity calculation means 121 (step A6) . 
The reference image's small region feature quantity is supplied 
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as input (stepA7) . The small region similarity calculation means 

121 calculates a small region similarity, i.e. , a feature quantity 
similarity between feature quantities obtained by comparing the 
input reference image's small region feature quantity with the 

5 inquiry image's small region feature quantity. The small region 
similarity calculation means 121 outputs the calculated small 
region similarity to the image similarity calculation means 122 
(step A8) . The image similarity calculation means 122 then 
weights the small region similarity using a weight value for each 

10 small region found from the local region weight value calculated 
by the local region weight calculation means 14 at step A3. The 
image similarity calculation means 122 calculates an image 
similarity between the inquiry image and the reference image (step 
A9) . In this manner, the use of the local region weight value 

15 makes it possible to eliminate the effect of a highly possibly 
edited region. Lastly, the image similarity calculation means 

122 outputs the calculated image similarity (step AlO) . 
[0048] 

The first embodiment of the present invention supplies a 
20 similarity for each of small regions of the divided image with 
a weight calculated in consideration for a probability model of 
editing the image to calculate the image similarity. 
Consequently, the image similarity calculation can be free from 
the effect of a highly possibly edited region. It is possible 
25 to correctly measure a similarity between the state of the inquiry 
image before editing and the reference image. It is possible to 
yield a large value for the image similarity between the edited 
image and the original image. Further, the first embodiment of 
the present invention needs no process for detecting an edited 
30 region from the inquiry image. A local region weight value is 
calculated from the learning image or the device characteristic. 
Accordingly, another process can be used to find the local region 
weight value in advance, making it possible to fast calculate the 



23 



image similarity. 
[0049] 

According to the embodiment, the image similarity 
calculation means 122 calculates an image similarity in 
5 consideration for a probability model of editing an image 

estimated by the editing probability model estimation means 13, 
specifically, in consideration for a probability model of editing 
the image for each local region. In addition, the small region 
similarity calculation means 121 may compare the input reference 

10 image's small region feature quantity with the inquiry image's 
small region feature quantity in consideration for a probability 
model and calculate a small region similarity, i.e. , a similarity 
between feature quantities for small regions. For example, it 
may be preferable to provide a highly possibly edited small region 

15 with a process of adjusting the small region similarity to a small 
value. It should be noted that the small region similarity 
calculation means 121 may calculate a small region similarity in 
consideration for the probability model. This also applies to 
the other embodiments to be described below. 

20 [0050] 

The editing probability model estimation means 13 need not 
calculate the local region editing probability at the timing 
corresponding to input of each inquiry image. For example, the 
editing probability model estimation means 13 may calculate the 

25 local region editing probability at the time of installing an 
apparatus or a system to which the present invention is applied, 
or at a later periodic time point. The editing probability model 
estimation means 13 maintains a calculation result. When an 
inquiry image is input, the editing probability model estimation 

30 means 13 outputs the maintained local region editing probability 
to the local region weight calculation means 14. This also 
applies to editing probability model estimation means according 
to the other embodiments to be described later. 
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[0051] 

Accordingly, it may be preferable to previously calculate 
the local region editing probability and store the calculated 
local region editing probability in storage means such as ROM. 
5 FIG. 4 is a block diagram showing a modification of the first 
embodiment. According to the modification, there is provided a 
storage means to store local region editing probabilities instead 
of the editing probability model estimation means 13 . 
[0052] 

10 According to the configuration as shown in FIG. 4, editing 

probability model storage means 139 stores the local region 
editing probability. It is preferable that the local region 
corresponds to a small region divided by the image division means 
111. In addition, the local region may correspond to any of 

15 multiple smaller regions resulting from further dividing a small 
region divided by the image division means 111. Moreover, the 
local region may correspond to each element of the image. 
Furthermore, the local region may correspond to any of regions 
divided in any shapes and sizes different from those for the small 

20 region division method for the image division means 111. The 
local region editing probability may be stored not only as a value 
commonly used for all images, but also as different values 
according to types of images to be used such as news images, sports 
images , and variety images . In the latter case , a user can specify 

25 an inquiry image type to use the local region editing probability 
corresponding to the inquiry image type. 
[0053] 

FIG. 5 is an explanatory diagram showing an example of local 
region editing probability stored in the editing probability 
30 model storage means 139 . In this example , the editing probability 
model storage means 139 stores probability of editing each of 
local regions divided from an image corresponding to the types 
of news , sports , and variety images . The news or sports image 
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is highly possibly edited at the top left, top right, and bottom 
of the image. Therefore, large values are assigned to local 
region editing probabilities for the corresponding local regions . 
The variety image also contains large values for probabilities 
5 of applying the editing process at the center of the image. 
[0054] 

The blocks other than the editing probability model storage 
means 139 in FIG. 4 have the same configurations and operations 
as those of the blocks shown in FIG. 1. This modification 
10 eliminates the need to provide the editing probability model 
estimation means 13 having the calculation function. 
[0055] 

It may be preferable to store local region editing 
probabilities in a rewritable storage means such as flash memory 

15 and enable local region editing probabilities to be changed. FIG . 
6 is a block diagram showing another modification of the first 
embodiment. According to the modification, there is provided a 
rewritable storage means to store local region editing 
probabilities instead of the editing probability model estimation 

20 means 13 . 
[0056] 

According to the configuration shown in FIG. 6, the editing 
probability model estimation means 13 calculates local region 
editing probability, i.e., probability of an editing process 

25 applied to each local region in an image from an edited learning 
image or characteristics of a device to perform the editing 
process supplied as additional input on an as-needed basis. The 
editing probability model estimation means 13 outputs the 
calculated local region editing probability to an editing 

30 probability model update means 149. The editing probability 
model estimation means 13 is supplied with a learning image or 
characteristics of a device to perform the editing process 
periodically or at occurrence of an event that may greatly change 
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the editing probability model, for example. 
[0057] 

The editing probability model update means 149 reads a local 
region editing probability prestored in the editing probability 
5 model storage means 139. The editing probability model update 
means 149 updates the local region editing probability using a 
local region editing probability read from the editing 
probability model storage means 139 and a local region editing 
probability input from the editing probability model estimation 

10 means 13. The editing probability model update means 149 stores 
the updated local region editing probability in the editing 
probability model storage means 139. In addition, the editing 
probability model update means 149 may update the local region 
editing probability by storing a new local region editing 

15 probability read from the editing probability model storage means 
139 unchangedly in the editing probability model storage means 
139. 
[0058] 

The bloclcs other than the editing probability model update 
20 means 149 and the editing probability model storage means 139 
shown in FIG. 6 have the same configurations and operations as 
those of the blocks shown in FIG. 1. This modification can 
maintain proper contents of the local region editing probability 
stored in the editing probability model storage means 139. 
25 [0059] 

[Second Embodiment] 

The second embodiment of the present invention will be 
described in further detail with reference to the accompanying 
drawings. As shown in a bloclc diagram of FIG. 7, the second 
30 embodiment differs from the first embodiment as follows. The 
second embodiment of the image similarity calculation system 
according to the present invention replaces the image similarity 
calculation portion 12 in FIG. 1 according to the first embodiment 
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with an image similarity calculation portion 22 . The second 
embodiment includes a reference image group small region feature 
quantity storage portion 25 and a retrieval result output means 
26. The second embodiment provides an image similarity 
5 calculation system having the reference image group small region 
feature quantity storage portion 25 and the retrieval result 
output means 26. Such system implements ah image retrieval 
system. 
[0060] 

10 The reference image group small region feature quantity 

storage portion 25 stores small region feature quantities for 
prestored multiple reference images. The reference image group 
small region feature quantity storage portion 25 stores small 
region feature quantities for respective reference images . It 

15 is preferable that the small region concerning each of these small 
region feature quantities corresponds to an inquiry image's small 
region divided by the image division means 111. In addition, the 
small region may be a division shaped and sized differently from 
the inquiry image's small region. 

20 [0061] 

The image similarity calculation portion 22 functionally 
includes small region similarity calculation means 221 and image 
similarity calculation means 222. 
[0062] 

25 The small region similarity calculation means 221 reads 

small region feature quantities for multiple reference images 
from the reference image group small region feature quantity 
storage portion 25 on an image basis . The small region similarity 
calculation means 221 compares the read small region feature 

30 quantity for each of the reference images with the inquiry image ' s 
small region feature quantity input from the small region feature 
quantity extraction means 112. The small region similarity 
calculation means 221 calculates a small region similarity, i.e. , 
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a feature quantity similarity for each small region. The small 
region similarity calculation means 221 outputs the calculated 
small region similarity for each reference image in the reference 
image group to the image similarity calculation means 222. 
5 [0063] 

Any method of calculating the small region similarity can 
be settled according to the small region feature quantity to be 
used. For example^ when the small region feature quantity to be 
used is a representative value such as an average value, a mode 

10 value, or a median value for the small region's color, there is 
a method of calculating the similarity based on a distance value 
and a correlation value in a color space being used. When the 
small region feature quantity to be used is an occurrence 
histogram for colors and edge elements, there is a method of 

15 calculating the similarity between histograms. 
[0064] 

The small region feature quantity to be used may be Dominant 
Color, Color Layout, Scalable Color, Color Structure, Edge 
Histogram, Homogeneous Texture, Texture Browsing, Region Shape, 

20 Contour Shape, Shape 3D, Parametric Motion, or Motion Activity 
specified in international standard ISO/IEC 15938-3. In this 
case, similarities can be calculated in accordance with the 
distance (similarity) calculation method recommended by ISO/IEC 
15938-3. 

25 [0065] 

The similarity may be a binary determination result such 
as 1 or 0 . When the edge element feature quantity is used as the 
small region feature quantity, for example, there is a method of 
setting the similarity to 1 if the comparing edge elements match 
30 or setting the similarity to 0 if the comparing edge elements do 
not match. Different image division methods may cause a 
discrepancy between a small region concerning the small region 
feature quantity for the inquiry image input from the small region 
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feature quantity extraction means 112 and a small region 
concerning the small region feature quantity for the reference 
image read from the reference image group small region feature 
quantity storage portion 25. To solve this problem, it is a good 
5 practice to use either image's small region feature quantity to 
complement the feature quantity corresponding to the other 
image's small region and calculate a similarity. For example, 
a small region concerning the reference image's small region 
feature quantity may result from further dividing the small region 

10 concerning the inquiry image ' s small region feature quantity into 
smaller multiple regions. In this case, it is possible to 
generate a feature quantity corresponding to the inquiry image's 
small region by calculating an average value of the reference 
image's multiple small region feature quantities. 

15 [0066] 

There may be a case where the small region division method 
concerning the reference image's small region feature quantity 
uses shapes and sizes completely different from those for the 
small region division method concerning the inquiry image ' s small 

20 region feature quantity. In this case, small region feature 
quantities of the reference image's small regions which overlap 
with the inquiry image ' s small regions are then weighted in 
proportion to overlapping areas and are totaled. In this manner, 
it is possible to generate the feature quantity corresponding to 

25 the inquiry image's small region. 
[0067] 

The image similarity calculation means 222 is supplied with 
the small region similarity for each reference image in the 
reference image group from the small region similarity 
30 calculation means 221. The image similarity calculation means 
222 weights the supplied small region similarity using a 
small-region-based weight value found from local region weight 
values supplied from the local' region weight calculation means 
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14. The image similarity calculation means 222 calculates an 
image similarity, i.e., a similarity for the overall image, from 
the resulting weighted small region similarities. The image 
similarity calculation means 222 outputs the calculated image 
5 similarity for each reference image in the reference image group 
to the retrieval result output means 26. 
[0068] 

Methods of calculating the image similarity include finding 
a total sum of weighted small region similarities for all small 

10 regions or finding an average value thereof. If the local region 
of the local region weight value input from the local region weight 
calculation means 14 corresponds to a small region of the small 
region similarity input from the small region similarity 
calculation means 121, the local region weight value can be 

15 directly used as a weight value for each small region. When the 
small region of the small region similarity does not correspond 
to the local region of the local region weight value, it is a good 
practice to use the local region weight value to complement the 
weight value corresponding to the small region of the small region 

20 similarity. For example, a local region of the local region 
weight value may result from further dividing a small region 
having the small region similarity into multiple smaller regions. 
In this case, it is possible to generate a weight value 
corresponding to the small region of the small region similarity 

25 by calculating an average value for multiple local region weight 
values . 
[0069] 

There may be a case where the local region division method 
concerning local region weight values uses shapes and sizes 
30 completely different from those for the small region division 
method concerning small region similarities. In this case, 
weight values of the multiple local regions which overlap with 
the small regions are then weighted in proportion to overlapping 
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areas and are totaled. In this manner, it is possible to generate 
the weight value corresponding to the small region of the small 
region similarity . 
[0070] 

5 The retrieval result output means 26 operates based on image 

similarities for reference images in the reference image group 
supplied from the image similarity calculation means 222 and 
outputs a retrieval result for the inquiry image from the 
reference image group. There is a retrieval method of outputting 

10 a reference image having the largest image similarity out of all 
reference images in the reference image group. Further, it may 
be preferable to compare image similarities for reference images 
in the reference image group with a predetermined threshold value 
and output all reference images whose image similarities are equal 

15 to or greater than the threshold value. 
[0071] 

Operations of the second embodiment will now be described 
with reference to the bloc]c diagram of FIG. 2 and a flowchart of 
FIG. 7. 
20 [0072] 

As the editing probability model estimation means 13 is 
supplied with a learning image or a device characteristic as input 
(step Bl) , it calculates a local region editing probability, i.e., 
a probability of editing each local region in the image, from the 

25 input learning image or device characteristic. The editing 
probability model estimation means 13 outputs the calculated 
local region editing probability to the local region weight 
calculation means 14 (step B2) . The local region weight 
calculation means 14 calculates a local region weight value, i.e., 

30 a weight value for each local region in the image, based on the 
local region editing probability (step B3) . 
[0073] 

As the image division means 111 is supplied with the edited 
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inquiry image as input (step B4) , it divides the input inquiry 
image into small regions and outputs the image divided into small 
region to the small region feature quantity extraction means 112 
(step B5) . The small region feature quantity extraction means 
5 112 extracts an inquiry image small region feature quantity^ i.e. , 
a feature quantity for each small region in the image divided into 
small regions. The small region feature quantity extraction 
means 112 outputs the extracted inquiry image small region feature 
quantity to the small region similarity calculation means 221 
10 (step B6) . The small region similarity calculation means 221 
reads the small region feature quantity for one reference image 
from the reference image group small region feature quantity 
storage portion 25 (step B7) . 
[0074] 

15 The small region similarity calculation means 221 compares 

the small region feature quantity of the one read reference image 
with the inquiry image small region feature quantity to calculate 
a small region similarity, i.e., a feature quantity similarity 
for each small region. The small region similarity calculation 

20 means 221 outputs the calculated small region similarity to the 
image similarity calculation means 222 (step B8) . The image 
similarity calculation means 222 then weights the small region 
similarity using a weight value for each small region found from 
the local region weight value calculated by the local region 

25 weight calculation means 14 at step B3 . The image similarity 
calculation means 222 calculates an image similarity between the 
one read reference image and the inquiry image (step B9) . The 
process checks whether or not image similarities are calculated 
for all reference images stored in the reference image group small 

30 region feature quantity storage portion 25 (step BIO). When there 
remains a reference image whose image similarity is not calculated, 
the process returns to step B7 . When image similarities are 
calculated for all the reference images, the process proceeds to 
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step Bll. Finally, the retrieval result output means 26 outputs 
a retrieval result for the inquiry image from the reference image 
group based on the image similarities for all the reference images 
(step Bll) . 
5 [00751 

The second embodiment of the present invention retrieves 
images using the image similarity calculation method having the 
effect according to the first embodiment of the present invention. 
The second embodiment retrieves images using the image similarity 

10 calculated by eliminating effects of highly possibly edited 
regions. Accordingly, the second embodiment can accurately 
retrieve the original image from edited images. Similar to the 
first embodiment, the second embodiment requires no process to 
detect an edited region from the inquiry image and makes it 

15 possible to previously find the local region weight value, thus 
enabling fast image retrieval. 
[0076] 

The modifications of the first embodiment can be also 
applied to the second embodiment. That is, the editing 
20 probability model storage means 13 9 may be provided instead of 
the editing probability model estimation means 13 . It may be 
preferable to provide the editing probability model update means 
149 and the editing probability model storage means 139. 
[0077] 

25 [Third Embodiment] 

The third embodiment of the present invention will be 
described in further detail with reference to the accompanying 
drawings. As shown in a bloclc diagram of FIG. 9, the third 
embodiment of the image similarity calculation system according 

30 to the present invention is composed of the feature quantity 
extraction portion 11, an image similarity calculation portion 
32, an editing region detection means (an example of editing 
probability calculation means) 33, and a local region weight 
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calculation means 34. The feature quantity extraction portion 
11 is the same as that provided for the first embodiment. 
[0078] 

The editing region detection means 33 uses a method of 
5 automatically detecting edited regions such as the ticker 

detection method for an edited inquiry image supplied as input. 
Using this method, the editing region detection means 33 
calculates a local region editing probability, i.e., a 
probability that an editing process was applied to each local 

10 region in the inquiry image. The editing region detection means 
33 outputs the calculated local region editing probability to the 
local region weight calculation means 34. The local region 
editing probability may have any probability value or a binary 
value 1 or 0. The local region may correspond to any of multiple 

15 smaller regions resulting from further dividing a small region 
divided by the image division means 111. Moreover, the local 
region may correspond to each element of the image. Furthermore, 
the local region may correspond to any of regions divided in any 
shapes and sizes different from those for the small region 

20 division method for the image division means 111. 
[0079] 

The local region weight calculation means 34 calculates a 
local region weight value based on the local region editing 
probability supplied from the editing region detection means 33 . 

25 The local region weight value quantifies how effectively the 
feature quantity extracted from local regions of an image works 
for calculation of the similarity for the overall image. The 
local region weight calculation means 34 outputs the calculated 
local region weight value to the image similarity calculation 

30 means 322. It is preferable that the local region corresponds 
to a small region divided by the image division means 111. In 
addition, the local region may correspond to any of multiple 
smaller regions resulting from further dividing a small region 
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divided by the image division means 111. Moreover, the local 
region may correspond to each element of the image. Furthermore, 
the local region may correspond to any of regions divided in any 
shapes and sizes different from those for the small region 
5 division method for the image division means 111. There is an 
example of the method for calculating the local region weight 
value as follows . 

Local region weight value = 1 - local region editing 
probability 

10 The present invention is not limited thereto. During the 

image similarity calculation process, it may be preferable to use 
the other calculation methods capable of eliminating the effect 
of a highly possibly edited region. The editing region detection 
means 33 may supply local region editing probabilities as binary 

15 probability values such as 1 and 0 . In this case, there is a method 
of integrating multiple local regions into a new local region. 
The method finds a local region weight value as a ratio of an area 
of the local region having the local region editing probability 
value of 0 before the integration included in the local region 

20 after the integration. 
[0080] 

The image similarity calculation portion 32 functionally 
includes a small region similarity calculation means 321 and an 
image similarity calculation means 322. 
25 [0081] 

The small region similarity calculation means 321 compares 
the inquiry image's small region feature quantity supplied from 
the small region feature quantity extraction means 112 with the 
small region feature quantity of the reference image supplied as 
30 input. The small region similarity calculation means 321 

calculates a small region similarity, i.e., a feature quantity 
similarity for each small region. The small region similarity 
calculation means 321 outputs the calculated small region 
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similarity to the image similarity calculation means 322 . Any 
method of calculating the small region similarity can be settled 
according to the small region feature quantity to be used. For 
example, when the small region feature quantity to be used is a 
5 representative value such as an average value, a mode value, or 
a median value for the small region's color, there is a method 
of calculating the similarity based on a distance value and a 
correlation value in a color space being used. 
[0082] 

10 When the small region feature quantity to be used is an 

occurrence histogram for colors and edge elements, there is a 
method of calculating the similarity between histograms. The 
small region feature quantity to be used may be Dominant Color, 
Color Layout, Scalable Color, Color Structure, Edge Histogram, 

15 Homogeneous Texture, Texture Browsing, Region Shape, Contour 
Shape, Shape 3D, Parametric Motion, or Motion Activity specified 
in ISO/IEC 15938—3. In this case, similarities can be calculated 
in accordance with the distance (similarity) calculation method 
recommended by ISO/IEC 15938-3. 

20 [0083] 

The similarity may be a binary determination result such 
as 1 or 0. When the edge element feature quantity is used as the 
small region feature quantity, for example, there is a method of 
setting the similarity to 1 if the comparing edge elements match 

25 or setting the similarity to 0 if the comparing edge elements do 
not match. It is preferable that a small region concerning the 
small region feature quantity for the reference image supplied 
as input corresponds to a small region concerning the small region 
feature quantity for the inquiry image supplied from the small 

30 region feature quantity extraction means 112. This is because 
corresponding small regions can be directly compared with each 
other. Different image division methods may cause a discrepancy 
between the inquiry image ' s small region and the reference image ' s 



37 



small region. To solve this problem, it is a good practice to 
use either image's small region feature quantity to complement 
the feature quantity corresponding to the other image's small 
region and calculate a similarity. For example, a small region 
5 concerning the reference image ' s small region feature quantity 
may result from further dividing the small region concerning the 
inquiry image's small region feature quantity into smaller 
multiple regions. In this case, it is possible to generate a 
feature quantity corresponding to the inquiry image's small 
10 region by calculating an average value of the reference image's 
multiple small region feature quantities . 
[0084] 

There may be a case where the small region division method 
concerning the reference image's small region feature quantity 

15 uses shapes and sizes completely different from those for the 
small region division method concerning the inquiry image' s small 
region feature quantity. In this case, small region feature 
quantities of the reference image's small regions which overlap 
with the inquiry image's small regions are then weighted in 

20 proportion to overlapping areas and are totaled. In this manner, 
it is possible to generate the feature quantity corresponding to 
the inquiry image's small region. 
[0085] 

The small region similarity calculation means 321 inputs 
25 the small region similarity. The image similarity calculation 
means 322 weights this small region similarity using a weight 
value for each small region found from a local region weight value 
input from the local region weight calculation means 34. The 
image similarity calculation means 322 calculates an image 
30 similarity, i.e., a similarity for the overall image, from the 
resulting weighted small region similarities and outputs the 
calculated image similarity. Methods of calculating the image 
similarity include finding a total sum of weighted small region 
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similarities for all small regions or finding an average value 

thereof . 

[0086] 

If the local region of the local region weight value input 
5 from the local region weight calculation means 34 corresponds to 
a small region of the small region similarity input from the small 
region similarity calculation means 321, the local region weight 
value can be directly used as a weight value for each small region. 
When the small region of the small region similarity does not 

10 correspond to the local region of the local region weight value, 
it is a good practice to use the local region weight value to 
complement the weight value corresponding to the small region of 
the small region similarity. For example, a local region of the 
local region weight value may result from further dividing a small 

15 region of the small region similarity into multiple smaller 

regions. In this case, it is possible to generate a weight value 
corresponding to the small region of the small region similarity 
by calculating an average value for multiple local region weight 
values . 

20 [0087] 

There may be a case where the local region division method 
concerning local region weight values uses shapes and sizes 
completely different from those for the small region division 
method concerning small region similarities. In this case, 

25 weight values of the multiple local regions which overlap with 
the small regions are then weighted in proportion to overlapping 
areas and are totaled. In this manner, it is possible to generate 
the weight value corresponding to the small region of the small 
region similarity . 

30 [0088] 

Similar to the first embodiment, the third embodiment 
calculates an image similarity by eliminating the effect of a 
highly possibly edited region. Further, the third embodiment 
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calculates a local region weight value based on the local region 
editing probability directly calculated from the inquiry image. 
The first embodiment heuristically finds a local region weight 
value as an average value from the learning image or the device 
5 characteristic. Differently from the first embodiment, the 

third embodiment calculates a local region weight value for each 
inquiry image, making it possible to calculate a larger value for 
the image similarity between the edited image and the original 
image. Further, the third embodiment requires no learning image 
10 or device characteristic as input. 
[0089] 

[ Fourth Embodiment ] 

The fourth embodiment of the present invention will be 
described in further detail with reference to the accompanying 

15 drawings. As shown in a block diagram of FIG. 10, the fourth 
embodiment differs from the third embodiment as follows. The 
fourth embodiment of the image similarity calculation system 
according to the present invention replaces the image similarity 
calculation portion 32 shown in FIG. 9 according to the third 

20 embodiment with an image similarity calculation portion 42. The 
fourth embodiment provides the reference image group small region 
feature quantity storage portion 25 and the retrieval result 
output means 26. The reference image group small region feature 
quantity storage portion 2 5 and the retrieval result output means 

25 2 6 are the same as those provided for the second embodiment. 
[0090] 

The image similarity calculation portion 42 functionally 
includes a small region similarity calculation means 421 and an 
image similarity calculation means 422 . 
30 [0091] 

The small region similarity calculation means 421 reads 
small region feature quantities for multiple reference images 
from the reference image group small region feature quantity 
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storage portion 25 on an image basis . The small region similarity 
calculation means 421 compares the read small region feature 
quantity for each of the reference images with the inquiry image ' s 
small region feature quantity input from the small region feature 
5 quantity extraction means 112. The small region similarity 
calculation means 421 calculates a small region similarity, i.e. , 
a feature quantity similarity for each small region. The small 
region similarity calculation means 421 outputs the calculated 
small region similarity for each reference image in the reference 

10 image group to the image similarity calculation means 422. Any 
method of calculating the small region similarity can be settled 
according to the small region feature quantity to be used. For 
example, when the small region feature quantity to be used is a 
representative value such as an average value, a mode value, or 

15 a median value for the small region's color, there is a method 
of calculating the similarity based on a distance value and a 
correlation value in a color space being used. 
[0092] 

When the small region feature quantity to be used is an 
20 occurrence histogram for colors and edge elements, there is a 
method of calculating the similarity between histograms. The 
small region feature quantity to be used may be Dominant Color, 
Color Layout, Scalable Color, Color Structure, Edge Histogram, 
Homogeneous Texture, Texture Browsing, Region Shape, Contour 
25 Shape, Shape 3D, Parametric Motion, or Motion Activity specified 
in international standard ISO/IEC 15938-3. In this case, 
similarities can be calculated in accordance with the distance 
(similarity) calculation method recommended by ISO/IEC 15938-3. 
[0093] 

30 The similarity may be a binary determination result such 

as 1 or 0 . When the edge element feature quantity is used as the 
small region feature quantity, for example, there is a method of 
setting the similarity to 1 if the comparing edge elements match 
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or setting the similarity to 0 if the comparing edge elements do 
not match. Different image division methods may cause a 
discrepancy between a small region concerning the small region 
feature quantity for the inquiry image input from the small region 
5 feature quantity extraction means 112 and a small region 

concerning the small region feature quantity for the reference 
image read from the reference image group small region feature 
quantity storage portion 25. To solve this problem, it is a good 
practice to use either image's small region feature quantity to 

10 complement the feature quantity corresponding to the other 

image's small region and calculate a similarity. For example, 
a small region concerning the reference image's small region 
feature quantity may result from further dividing the small region 
concerning the inquiry image* s small region feature quantity into 

15 smaller multiple regions. In this case, it is possible to 

generate a feature quantity corresponding to the inquiry image's 
small region by calculating an average value of the reference 
image's multiple small region feature quantities. 
[0094] 

20 There may be a case where the small region division method 

concerning the reference image's small region feature quantity 
uses shapes and sizes completely different from those for the 
small region division method concerning the inquiry image ' s small 
region feature quantity. In this case, small region feature 

25 quantities of the reference image's small regions which overlap 
with the inquiry image's small regions are then weighted in 
proportion to overlapping areas and are totaled. In this manner, 
it is possible to generate the feature quantity corresponding to 
the inquiry image's small region. 

30 [0095] 

The image similarity calculation means 422 is supplied with 

- * 

the small region similarity for each reference image in the 
reference image group from the small region similarity 
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calculation means 421. The image similarity calculation means 
422 weights the supplied small region similarity using a 
small-region-based weight value found from local region weight 
values supplied from the local region weight calculation means 
5 34. The image similarity calculation means 422 calculates an 
image similarity, i.e. , a similarity for the overall image, from 
the resulting weighted small region similarities. The image 
similarity calculation means 422 outputs the calculated image 
similarity for each reference image in the reference image group 
10 to the retrieval result output means 26. 
[0096] 

Methods of calculating the image similarity include finding 
a total sum of weighted small region similarities for all small 
regions or finding an average value thereof. If the local region 

15 of the local region weight value input from the local region weight 
calculation means 34 corresponds to a small region of the small 
region similarity input from the small region similarity 
calculation means 421, the local region weight value can be 
directly used as a weight value for each small region. When the 

20 small region of the small region similarity does not correspond 
to the local region of the local region weight value, it is a good 
practice to use the local region weight value to complement the 
weight value corresponding to the small region of the small region 
similarity. For example, a local region of the local region 

25 weight value may result from further dividing a small region of 
the small region similarity into multiple smaller regions. In 
this case, it is possible to generate a weight value corresponding 
to the small region of the small region similarity by calculating 
an average value for multiple local region weight values . 

30 [0097] 

Further, there may be a case where the local region division 
method concerning local region weight values uses shapes and sizes 
completely different from those for the small region division 
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method concerning small region similarities. In this case, 
weight values of the multiple local regions which overlap with 
the small regions are then weighted in proportion to overlapping 
areas. and are totaled. In this manner, it is possible to generate 
5 the weight value corresponding to the small region of the small 
region similarity . 
[0098] 

Similar to the second embodiment, the retrieval result 
output means 2 6 outputs a retrieval result. 
10 [0099] 

The fourth embodiment retrieves images using the image 
similarity calculation method having the effect according to the 
third embodiment of the present invention. Differently from the 
second embodiment, the fourth embodiment calculates a local 
15 region weight value for each inquiry image, making it possible 
to more precisely retrieve the original image from an edited image. 
Similar to the third embodiment, the fourth embodiment requires 
no learning image or device characteristic as input. 
[0100] 

20 [Fifth Embodiment] 

The fifth embodiment of the present invention will be 
described in further detail with reference to the accompanying 
drawings. As shown in a block diagram of FIG. 11, the fifth 
embodiment of the image similarity calculation system according 

25 to the present invention is composed of the feature quantity 
extraction portion 11, an image similarity calculation portion 
52, an editing probability model estimation means 53, and a match 
determination threshold calculation means 54. The feature 
quantity extraction portion 11 is the same as that provided for 

30 the first embodiment of the present invention. 
[0101] 

The editing probability model estimation means (an example 
of editing probability calculation means) 53 calculates local 
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region editing probability or editing- targeted local region count 
probability distribution from an edited learning image or 
characteristics of a device to perform the editing process 
supplied as input. The local region editing probability 
5 signifies probability of editing each local region in the image. 
The editing-targeted local region count probability distribution 
signifies probability distribution of the number of local regions 
in the image to be edited. The editing probability model 
estimation means 53 outputs the calculated local region editing 
10 probability or editing- targeted local region count probability 
distribution to the match determination threshold calculation 
means 54. 
[0102] 

Examples of the device characteristic include editing 
15 patterns applied to an image by the editing device and information 
about the frequency of using each editing patterns. It is 
preferable that the local region corresponds to a small region 
divided by the image division means 111. In addition, the local 
region may correspond to any of multiple smaller regions resulting 
20 from further dividing a small region divided by the image division 
means 111. Moreover, the local region may correspond to each 
element of the image. Furthermore, the local region may 
correspond to any of regions divided in any shapes and sizes 
different from those for the small region division method for the 
25 image division means 111. 
[0103] 

When an edited learning image is input as a learning image, 
the local region editing probability is measured by specifying 
regions that an editing process was applied in the learning image. 
30 Methods of specifying edited regions may include not only manual 
methods, but also methods of automatically detecting editing 
processes such as the ticker detection method. Characteristics 
of a device to perform editing processes may be supplied as input. 
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In such case, there is available the method of measuring the local 
region editing probability based on editing patterns applied to 
an image by the device and based on information about the frequency 
of using each editing patterns. 
5 [0104] 

The match determination threshold calculation means 54 
operates as follows based on the local region editing probability 
or the editing-targeted local region count probability 
distribution supplied from the editing probability model 

10 estimation means 53 . The match determination threshold 

calculation means 54 calculates a match determination threshold 
for the overall image's similarity found by an image similarity 
calculation means 522 to be described later. The match 
determination threshold represents an optimum threshold value for 

15 determining whether or not two compared images match. The match 
determination threshold calculation means 54 outputs the 
calculated match determination threshold to the image similarity 
calculation means 522. There is a method of calculating an 
optimum threshold value. The method estimates occurrence 

20 probability distribution of the overall image's similarity. The 
method estimates the occurrence probability distribution of the 
overall image's similarity found from similarities of the image's 
small regions between an edited image with partially varied 
patterns and the original image based on the local region editing 

25 probability or the editing- targeted local region count 

probability distribution. Based on the estimated occurrence 
probability distribution of the overall image's similarity, the 
method finds a threshold value that the probability of determining 
a mismatch (discrepancy) between the edited image and the original 

30 image is small enough. The method provides for effects of pattern 
changes due to the editing process while estimating the occurrence 
probability distribution of the overall image's similarity. In 
addition, it may be preferable to provide for effects of feature 
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quantity variations due to noise occurrence resulting from 

re-encoding the image . 

[0105] 

The image similarity calculation portion 52 functionally 
5 includes a small region similarity calculation means 521 and an 
image similarity calculation means 522. 
[0106] 

The small region similarity calculation means 521 compares 
the small region feature quantity of an inquiry image supplied 

10 from the small region feature quantity extraction means 112 with 
that of a reference image supplied as input. The small region 
similarity calculation means 521 calculates a small region 
similarity, i.e., a feature quantity similarity for each small 
region. The small region similarity calculation means 521 

15 outputs the calculated small region similarity to the image 

similarity calculation means 522. Any method of calculating the 
small region similarity may be settled according to the small 
region feature quantity to be used. For example, when the small 
region feature quantity to be used is a representative value such 

20 as an average value, a mode value, or a median value for the small 
region's color, there is a method of calculating the similarity 
based on a distance value and a correlation value in a color space 
being used. 
[0107] 

25 When the small region feature quantity to be used is an 

occurrence histogram for colors and edge elements, there is a 
method of calculating the similarity between histograms. The 
small region feature quantity to be used may be Dominant Color, 
Color Layout, Scalable Color, Color Structure, Edge Histogram, 

30 Homogeneous Texture, Texture Browsing, Region Shape, Contour 
Shape, Shape 3D, Parametric Motion, or Motion Activity specified 
in ISO/IEC 15938-3. In this case, similarities can be calculated 
in accordance with the distance (similarity) calculation method 
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recommended by ISO/IEC 15938-3. 
[0108] 

The similarity may be a binary determination result such 
as 1 or 0. When the edge element feature quantity is used as the 
5 small region feature quantity, for example, there is a method of 
setting the similarity to 1 if the comparing edge elements match 
or setting the similarity to 0 if the comparing edge elements do 
not match. It is preferable that a small region concerning the 
small region feature quantity for the reference image supplied 

10 as input corresponds to a small region concerning the small region 
feature quantity for the inquiry image supplied from the small 
region feature quantity extraction means 112. This is because 
corresponding small regions can be directly compared with each 
other. Different image division methods may cause a discrepancy 

15 between the inquiry image ' s small region and the reference image ' s 
small region. To solve this problem, it is a good practice to 
use either imagers small region feature quantity to complement 
the feature quantity corresponding to the other image's small 
region and calculate a similarity. For example, a small region 

20 concerning the reference image's small region feature quantity 
may result from further dividing the small region concerning the 
inquiry image's small region feature quantity into smaller 
multiple regions. In this case, it is possible to generate a 
feature quantity corresponding to the inquiry image's small 

25 region by calculating an average value of the reference image's 
multiple small region feature quantities. 
[0109] 

There may be a case where the small region division method 
concerning the reference image's small region feature quantity 

30 uses shapes and sizes completely different from those for the 
small region division method concerning the inquiry image ' s small 
region feature quantity. In this case, small region feature 
quantities of the reference image's small regions which overlap 
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with the inquiry image * s small regions are then weighted in 
proportion to overlapping areas and are totaled. In this manner, 
it is possible to generate the feature quantity corresponding to 
the inquiry image ' s small region . 
5 [0110] 

The image similarity calculation means 522 calculates the 
overall image's similarity from small region similarities 
supplied from the small region similarity calculation means 521. 
The image similarity calculation means 522 modifies the 

10 calculated overall image's similarity using the match 

determination threshold supplied from the match determination 
threshold calculation means 54. The image similarity 
calculation means 522 outputs the modified similarity as an image 
similarity. Methods of calculating the overall image's 

15 similarity include finding a total sum of small region 

similarities for all small regions or finding an average value 
thereof. There is a method of modifying the overall image's 
similarity using the match determination threshold. The method 
modifies the similarity to 1 when the image similarity is equal 

20 to or greater than the match determination threshold. The method 
modifies the similarity to 0 when the image similarity is smaller 
than the match determination threshold. 
[0111] 

Operations of the fifth embodiment will now be described 
25 with reference to the bloclc diagram of FIG. 11 and a flowchart 
of FIG. 12. 
[0112] 

As the editing probability model estimation means 53 is 
supplied with the learning image or the device characteristic as 
30 input (step CI) , it calculates local region editing probability 
or editing-targeted local region count probability distribution 
from the supplied learning image or device characteristic. The 
local region editing probability signifies probability of editing 
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each local region in the image. The editing- targeted local region 
count probability distribution signifies probability 
distribution of the number of local regions in the image to be 
edited. The editing probability model estimation means 53 
5 outputs the calculated local region editing probability or 

editing-targeted local region count probability distribution to 
the match determination threshold calculation means 54 (step C2) . 
Based on the local region editing probability or the 
editing- targeted local region count probability distribution, 
10 the match determination threshold calculation means 54 calculates 
a match determination threshold, i.e., a threshold value for 
determining a match between images (step C3) . 
[0113] 

As the image division means 111 is supplied with the edited 

15 inquiry image as input (step C4) , it divides the input inquiry 
image into small regions. The image division means 111 outputs 
the image divided into small regions to the small region feature 
quantity extraction means 112 (step C5) . The small region feature 
quantity extraction means 112 extracts an inquiry image small 

20 region feature quantity, i.e. , a feature quantity for each of the 
divided small regions in the image. The small region feature 
quantity extraction means 112 outputs the extracted inquiry image 
small region feature quantity to the small region similarity 
calculation means 521 (step C6) . The small region similarity 

25 calculation means 521 is supplied with the reference image ' s small 
region feature quantity as input (step C7) . The small region 
similarity calculation means 521 compares the supplied reference 
image ' s small region feature quantity with the inquiry image small 
region feature quantity to calculate a small region similarity, 

30 i.e., a feature quantity similarity for each small region. The 
small region similarity calculation means 521 outputs the 
calculated small region similarity to the image similarity 
calculation means 522 (step C8) . The image similarity 
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calculation means 522 calculates the overall image's similarity 
from the small region similarities. The image similarity 
calculation means 522 modifies the overall image's similarity 
using the match determination threshold calculated by the match 
5 determination threshold calculation means 54 at step C3 to 

calculate an image similarity between the inquiry image and the 
reference image (step C9) . Finally, the image similarity 
calculation means 522 outputs the calculated image similarity 
(step CIO) . 
10 [0114] 

Based on a probability model for editing images, the fifth 
embodiment calculates an optimum threshold value for determining 
a match between two images to be compared. Based on this threshold 
value, the embodiment modifies a similarity between images to 

15 calculate the image similarity. The embodiment modifies the 

similarity based on the threshold value for determining a match 
between images. Accordingly, the embodiment can calculate an 
image similarity suited for determining whether or not an edited 
inquiry image is generated from the reference image. The fifth 

20 embodiment requires no process to detect an edited region from 
the inquiry image. The embodiment calculates the match 
determination threshold from a learning image or a device 
characteristic. Accordingly, it is possible to previously find 
the match determination threshold using another process and fast 

25 calculate the image similarity. 
[0115] 

The modifications of the first embodiment can be also 
applied to the fifth embodiment . That is , the editing probability 
model storage means may be provided instead of the editing 
30 probability model estimation means 53. It may be preferable to 
provide the editing probability model update means and the editing 
probability model storage means. 
[0116] 
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[Sixth Embodiment] 

The sixth embodiment of the present invention will be 
described in further detail with reference to the accompanying 
drawings. As shown in a block diagram of FIG. 13, the sixth 
5 embodiment differs from the fifth embodiment as follows. The 
sixth embodiment of the image similarity calculation system 
according to the present invention replaces the image similarity 
calculation portion 52 shown in FIG. 11 according to the fifth 
embodiment with an image similarity calculation portion 62 . The 

10 sixth embodiment includes the reference image group small region 
feature quantity storage portion 25 and the retrieval result 
output means 26. The reference image group small region feature 
quantity storage portion 25 and the retrieval result output means 
26 are the same as those provided for the second embodiment. 

15 [0117] 

The image similarity calculation portion 62 functionally 
includes a small region similarity calculation means 621 and an 
image similarity calculation means 622 . 
[0118] 

20 The small region similarity calculation means 621 reads 

small region feature quantities for multiple reference images 
from the reference image group small region feature quantity 
storage portion 25 on an image basis. The small region similarity 
calculation means 621 compares the read small region feature 

25 quantity for each of the reference images with the inquiry image ' s 
small region feature quantity input from the small region feature 
quantity extraction means 112. The small region similarity 
calculation means 621 calculates a small region similarity^ i.e. , 
a feature quantity similarity for each small region. The small 

30 region similarity calculation means 621 outputs the calculated 
small region similarity for each reference image in the reference 
image group to the image similarity calculation means 622. Any 
method of calculating the small region similarity can be settled 
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according to the small region feature quantity to be used. 
[0119] 

For example, when the small region feature quantity to be 
used is a representative value such as an average value, a mode 
5 value, or a median value for the small region's color, there is 
a method of calculating the similarity based on a distance value 
and a correlation value in a color space being used. When the 
small region feature quantity to be used is an occurrence 
histogram for colors and edge elements, there is a method of 
10 calculating the similarity between histograms. 
[0120] 

The small region feature quantity to be used may be Dominant 
Color, Color Layout, Scalable Color, Color Structure, Edge 
Histogram, Homogeneous Texture, Texture Browsing, Region Shape, 

15 Contour Shape, Shape 3D, Parametric Motion, or Motion Activity 
specified in international standard ISO/IEC 15938-3. In this 
case, similarities can be calculated in accordance with the 
distance (similarity) calculation method recommended by ISO/IEC 
15938-3. 

20 [0121] 

The similarity may be a binary determination result such 
as 1 or 0 . When the edge element feature quantity is used as the 
small region feature quantity, for example, there is a method of 
setting the similarity to 1 if the comparing edge elements match 

25 or setting the similarity to 0 if the comparing edge elements do 
not match. Different image division methods may cause a 
discrepancy between a small region concerning the small region 
feature quantity for the inquiry image input from the small region 
feature quantity extraction means 112 and a small region 

30 concerning the small region feature quantity for the reference 
image read from the reference image group small region feature 
quantity storage portion 25. To solve this problem, it is a good 
practice to use either image's small region feature quantity to 
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complement the feature quantity corresponding to the other 
image's small region and calculate a similarity. For example, 
a small region concerning the reference image's small region 
feature quantity may result from further dividing the small region 
concerning the inquiry image ' s small region feature quantity into 
smaller multiple regions. In this case, it is possible to 
generate a feature quantity corresponding to the inquiry image's 
small region by calculating an average value of the reference 
image's multiple small region feature quantities. 
[0122] 

There may be a case where the small region division method 
concerning the reference image's small region feature quantity 

* 

uses shapes and sizes completely different from those for the 
small region division method concerning the inquiry image ' s small 
region feature quantity. In this case, small region feature 
quantities of the reference image's small regions which overlap 
with the inquiry image's small regions are then weighted in 
proportion to overlapping areas and are totaled. - In this manner, 
it is possible to generate the feature quantity corresponding to 
the inquiry image's small region. 
[0123] 

The image similarity calculation means 622 calculates an 
overall image's similarity from small region similarities for the 
reference images in the reference image group supplied from the 
small region similarity calculation means 621. The image 
similarity calculation means 622 modifies the calculated overall 
image's similarity using the match determination threshold 
supplied from the match determination threshold calculation means 
54. The image similarity calculation means 622 outputs an image 
similarity, i.e., a similarity of reference images in the 
reference image group, to the retrieval result output means 26. 
Methods of calculating the overall image's similarity include 
finding a total sum of small region similarities for all small 



regions or finding an average value thereof. There is a method 
of modifying the overall image's similarity using the match 
determination threshold. The method modifies the similarity to 
1 when the image similarity is equal to or greater than the match 
5 determination threshold. The method modifies the similarity 
when the image similarity is smaller than the match determination 
threshold. 
[0124] 

Similar to the second embodiment, the retrieval result 
10 output means 26 outputs a retrieval result. 
[0125] 

Operations of the sixth embodiment will now be described 
with reference to the block diagram of FIG. 13 and a flowchart 
of FIG. 14. 
15 [0126] 

As the editing probability model estimation means 53 is 
supplied with the learning image or the device characteristic as 
input (step Dl) , it calculates local region editing probability 
or editing-targeted local region count probability distribution 

20 from the supplied learning image or device characteristic. The 
local region editing probability signifies probability of editing 
each local region in the image. The editing-targeted local region 
count probability distribution signifies probability 
distribution of the number of local regions in the image to be 

25 edited.. The editing probability model estimation means 53 
outputs the calculated local region editing probability or 
editing-targeted local region count probability distribution to 
the match determination threshold calculation means 54 (step D2) . 
Based on the local region editing probability or the 

30 editing- targeted local region count probability distribution, 
the match determination threshold calculation means 54 calculates 
a match determination threshold, i.e., a threshold value for 
determining a match between images (step D3) . 
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[0127] 

The image division means 111 is supplied with the edited 
inquiry image as input (step D4) . The image division means 111 
divides the input inquiry image into small regions. The image 
5 division means 111 outputs the image divided into small regions 
to the small region feature quantity extraction means 112 (step 
D5) . The small region feature quantity extraction means 112 
extracts an inquiry image small region feature quantity, i,e», 
a feature quantity for each of the divided small regions in the 

10 image. The small region feature quantity extraction means 112 
outputs the extracted inquiry image small region feature quantity 
to the small region similarity calculation means 621 (step D6) . 
The small region similarity calculation means 621 reads the small 
region feature quantity for one reference image from the reference 

15 image group small region feature quantity storage portion 2 5 (step 
D7) . The small region similarity calculation means 621 compares 
the small region feature quantity of the one read reference image 
with the inquiry image small region feature quantity to calculate 
a small region similarity, i.e., a feature quantity similarity 

2 0 for each small region. The small region similarity calculation 
means 621 outputs the calculated small region similarity to the 
image similarity calculation means 622 (step D8) . 
[0128] 

The image similarity calculation means 622 calculates the 
25 overall image's similarity from the small region similarities. 
The image similarity calculation means 622 modifies the overall 
image's similarity using the match determination threshold 
calculated by the match determination threshold calculation means 
54 at step D3 to calculate an image similarity between the one 
30 read reference image and the inquiry image (step D9) . The process 
checlcs whether or not image similarities are calculated for all 
reference images stored in the reference image group small region 
feature quantity storage portion 25 (step DIO) . When there 
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remains a reference image whose image similarity is not calculated, 
the process returns to step D7 . When image similarities are 
calculated for all the reference images, the process proceeds to 
step Dll. Finally, the retrieval result output means 26 outputs 
5 a retrieval result for the inquiry image from the reference image 
group based on the image similarities for all the reference images 

(step Dll) . 

[0129] 

The sixth embodiment retrieves images using the fifth 
10 embodiment that calculates an image similarity suited for 

determining whether or not an edited inquiry image is generated 
from the reference image. Even when the reference image group 
registers multiple original images for an edited inquiry image, 
the sixth embodiment can precisely retrieve all images determined 
15 to be the original image for the edited inquiry images . Similarly 
to the fifth embodiment, the sixth embodiment requires no process 
to detect an edited region from the inquiry image and makes it 
possible to previously find the match determination threshold, 
thus enabling fast image retrieval. 
20 [0130] 

The modifications of the first embodiment can be also 
applied to the sixth embodiment. That is, the editing probability 
model storage means may be provided instead of the editing 
probability model estimation means 53. It may be preferable to 
25 provide the editing probability model update means and the editing 
probability model storage means. 
[0131] 

[Seventh Embodiment] 

The seventh embodiment of the present invention will be 
30 described in further detail with reference to the accompanying 
drawings. As shown in a block diagram of FIG. 15, the seventh 
embodiment of the image similarity calculation system according 
to the present invention is functionally composed of the feature 
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quantity extraction portion 11, an image similarity calculation 
portion 72, an editing region detection means 73, and a match 
determination threshold calculation means 74. The feature 
quantity extraction portion 11 is the same as that provided for 
5 the first embodiment. 
[0132] 

The editing region detection means 73 uses a method of 
automatically detecting edited regions such as the ticker 
detection method for an edited inquiry image supplied as input. 

10 Using this method, the editing region detection means 73 

calculates local region editing probability or editing- targeted 
local region count probability distribution. The local region 
editing probability signifies a probability that an editing 
process was applied to each local region in the edited inquiry 

15 image. The editing- targeted local region count probability 

distribution signifies probability distribution of the number of 
local regions in the edited inquiry image that an editing process 
was applied. The editing region detection means 7 3 outputs the 
calculated local region editing probability or editing- targeted 

20 local region count probability distribution to the match 

determination threshold calculation means 74. The local region 
editing probability may have any probability value or a binary 
value 1 or 0 . The local region may correspond to any of multiple 
smaller regions resulting from further dividing a small region 

25 divided by the image division means 111. Moreover, the local 
region may correspond to each element of the image. Furthermore, 
the local region may correspond to any of regions divided in any 
shapes and sizes different from those for the small region 
division method for the image division means 111. 

30 [0133] 

The match determination threshold calculation means 74 
operates as follows based on the local region editing probability 
or the editing- targeted local region count probability 
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distribution supplied from the editing region detection means 73 . 
The match determination threshold calculation means 7 4 calculates 
a match determination threshold for the overall image's 
similarity found by an image similarity calculation means 722 to 
5 be described later. The match determination threshold 

represents an optimum threshold value for determining whether or 
not two compared images match. The match determination threshold 
calculation means 74 outputs the calculated match determination 
threshold to the image similarity calculation means 722. There 

10 is a method of calculating an optimum threshold value . The method 
estimates occurrence probability distribution of the overall 
image's similarity. The method estimates the occurrence 
probability distribution of the overall image's similarity found 
from similarities of the image's small regions between an edited 

15 image with partially varied patterns and the original image based 
on the local region editing probability or the editing- targeted 
local region count probability distribution. Based on the 
estimated occurrence probability distribution of the overall 
image's similarity, the methodv finds a threshold value that the 

20 probability of determining a mismatch (discrepancy) between the 
edited image and the original image is small enough. The method 
provides for effects of pattern changes due to the editing process 
while estimating the occurrence probability distribution of the 
overall image's similarity. In addition, it may be preferable 

25 to provide for effects of feature quantity variations due to noise 
occurrence resulting from re-encoding the image. 
[0134] 

The image similarity calculation portion 7 2 functionally 
includes a small region similarity calculation means 721 and an 
30 image similarity calculation means 722 . 
[0135] 

The small region similarity calculation means 721 compares 
the small region feature quantity of an inquiry image supplied 
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from the small region feature quantity extraction means 112 with 
that of a reference image supplied as input. The small region 
similarity calculation means 721 calculates a small region 
similarity, i.e., a feature quantity similarity for each small 
5 region. The small region similarity calculation means 721 
outputs the calculated small region similarity to the image 
similarity calculation means 722. Any method of calculating the 
small region similarity may be settled according to the small 
region feature quantity to be used. For example, when the small 

10 region feature quantity to be used is a representative value such 
as an average value, a mode value, or a median value for the small 
region's color, there is a method of calculating the similarity 
based on a distance value and a correlation value in a color space 
being used. When the small region feature quantity to be used 

15 is an occurrence histogram for colors and edge elements, there 
is a method of calculating the similarity between histograms. 
[0136] 

The small region feature quantity to be used may be Dominant 
Color, Color Layout, Scalable Color, Color Structure, Edge 

20 Histogram, Homogeneous Texture, Texture Browsing, Region Shape, 
Contour Shape, Shape 3D, Parametric Motion, or Motion Activity 
specified in ISO/IEC 15938-3. In this case, similarities can be 
calculated in accordance with the distance (similarity) 
calculation method recommended by ISO/IEC 15938-3. 

25 [0137] 

The similarity may be a binary determination result such 
as 1 or 0. When the edge element feature quantity is used as the 
small region feature quantity, for example, there is a method of • 
setting the similarity to 1 if the comparing edge elements match 
30 or setting the similarity to 0 if the comparing edge elements do 
not match. It is preferable that a small region concerning the 
small region feature quantity for the reference image supplied 
as input corresponds to a small region concerning the small region 
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feature quantity for the inquiry image supplied from the small 
region feature quantity extraction means 112. This is because 
corresponding small regions can be directly compared with each 
other. Different image division methods may cause a discrepancy 
5 between the inquiry image ' s small region and the reference image * s 
small region. To solve this problem, it is a good practice to 
use either image's small region feature quantity to complement 
the feature quantity corresponding to the other image's small 
region and calculate a similarity. For example, a small region 
10 concerning the reference image's small region feature quantity 
may result from further dividing the small region concerning the 

ft 

inquiry image's small region feature quantity into smaller 
multiple regions. In this case, it is possible to generate a 
feature quantity corresponding to the inquiry image's small 
15 region by calculating an average value of the reference image's 
multiple small region feature quantities. 
[0138] 

There may be a case where the small region division method 
concerning the reference image's small region feature quantity 

20 uses shapes and sizes completely different from those for the 
small region division method concerning the inquiry image ' s small 
region feature quantity. In this case, small region feature 
quantities of the reference image's small regions which overlap 
with the inquiry image's small regions are then weighted in 

25 proportion to overlapping areas and are totaled. In this manner, 
it is possible to generate the feature quantity corresponding to 
the inquiry image's small region. 
[0139] 

The image similarity calculation means 722 calculates the 
30 overall image's similarity from small region similarities 

supplied from the small region similarity calculation means 721. 
The image similarity calculation means 722 modifies the 
calculated overall image's similarity using the match 
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determination threshold supplied from the match determination 
threshold calculation means 74. The image similarity 
calculation means 722 outputs the modified similarity as an image 
similarity. Methods of calculating the overall image's 
5 similarity include finding a total sum of small region 

similarities for all small regions or finding an average value 
thereof. There is a method of modifying the overall image's 
similarity using the match determination threshold. The method 
modifies the similarity to 1 when the image similarity is equal 
10 to or greater than the match determination threshold. The method 
modifies the similarity to 0 when the image similarity is smaller 
than the match determination threshold. 
[0140] 

Similar to the fifth embodiment, the seventh embodiment 
15 calculates an image similarity suited for determining whether or 
not an edited inquiry image is generated from the reference image. 
In addition, the seventh embodiment calculates a match 
determination threshold based on the local region editing 
probability directly calculated from the inquiry image. The 
20 seventh embodiment calculates a match determination threshold for 
each inquiry image differently from the fifth embodiment that 
heuristically finds the match determination threshold as an 
average value from the learning image or the device characteristic. 
Accordingly, the seventh embodiment can more accurately find the 
25 image similarity suited for determining whether or not an edited 
inquiry image is generated from the reference image. The seventh 
embodiment requires no learning image or device characteristic 
as input. 
[0141] 

30 [Eighth iEmbodiment] 

The eighth embodiment of the present invention will be 
described in further detail with reference to the accompanying 
drawings. As shown in a block diagram of FIG. 16, the eighth 
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embodiment differs from the seventh embodiment as follows. The 
eighth embodiment of the image similarity calculation system 
according to the present invention replaces the image similarity 
calculation portion 72 shown in FIG. 15 according to the seventh 
5 embodiment with an image similarity calculation portion 82. The 
eighth embodiment provides the reference image group small region 
feature quantity storage portion 25 and the retrieval result 
output means 26. The reference image group small region feature 
quantity storage portion 25 and the retrieval result output means 
10 26 are the same as those provided for the second embodiment. 
[0142] 

The image similarity calculation portion 82 functionally 
includes a small region similarity calculation means 821 and an 
image similarity calculation means 822. 
15 [0143] 

The small region similarity calculation means 821 reads 
small region feature quantities for multiple reference images 
from the reference image group small region feature quantity 
storage portion 25 on an image basis . The small region similarity 

20 calculation means 821 compares the read small region feature 
quantity for each of the reference images with the inquiry image's 
small region feature quantity input from the small region feature 
quantity extraction means 112. The small region similarity 
calculation means 821 calculates a small region similarity, i.e. , 

25 a feature quantity similarity for each small region. The small 
region similarity calculation means 821 outputs the calculated 
small region similarity for each reference image in the reference 
image group to the image similarity calculation means 822. Any 
method of calculating the small region similarity can be settled 

30 according to the small region feature quantity to be used. For 
example, when the small region feature quantity to be used is a 
representative value such as an average value, a mode value, or 
a median value for the small region's color, there is a method 
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of calculating the similarity based on a distance value and a 
correlation value in a color space being used. When the small 
region feature quantity to be used is an occurrence histogram for 
colors and edge elements^ there is a method of calculating the 
5 similarity between histograms . 
[0144] 

The small region feature quantity to be used may be Dominant 
Color, Color Layout, Scalable Color, Color Structure, Edge 
Histogram, Homogeneous Texture, Texture Browsing, Region Shape, 

10 Contour Shape, Shape 3D, Parametric Motion, or Motion Activity 
specified in international standard ISO/IEC 15938-3. In this 
case, similarities can be calculated in accordance with the 
distance (similarity) calculation method recommended by ISO/IEC 
15938-3. 

15 [0145] 

The similarity may be a binary determination result such 
as 1 or 0 . When the edge element feature quantity is used as the 
small region feature quantity, for example, there is a method of 
setting the similarity to 1 if the comparing edge elements match 

20 or setting the similarity to 0 if the comparing edge elements do 
not match. Different image division methods may cause a 
discrepancy between a small region concerning the small region 
feature quantity for the inquiry image input from the small region 
feature quantity extraction means 112 and a small region 

25 concerning the small region feature quantity for the reference 
image read from the reference image group small region feature 
quantity storage portion 25. To solve this problem, it is a good 
practice to use either image's small region feature quantity to 
complement the feature quantity corresponding to the other 

30 image's small region and calculate a similarity. For example, 
a small region concerning the reference image's small region 
feature quantity may result from further dividing the small region 
concerning the inquiry image ' s small region feature quantity into 
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smaller multiple regions. In this case, it is possible to 
generate a feature quantity corresponding to the inquiry image's 
small region by calculating an average value of the reference 
image's multiple small region feature quantities. 
5 [0146] 

There may be a case where the small region division method 
concerning the reference image's small region feature quantity 
uses shapes and sizes completely different from those for the 
small region division method concerning the inquiry image ' s small 

10 region feature quantity. In this case, small region feature 
quantities of the reference image's small regions which overlap 
with the inquiry image's small regions are then weighted in 
proportion to overlapping areas and are totaled. In this manner, 
it is possible to generate the feature quantity corresponding to 

15 the inquiry image's small region. 
[0147] 

The image similarity calculaition means 822 calculates an 
overall image's similarity from small region similarities for the 
reference images in the reference image group supplied from the 

20 small region similarity calculation means 821. The image 

similarity calculation means 822 modifies the calculated overall 
image's similarity using the match determination threshold 
supplied from the match determination threshold calculation means 
74. The image similarity calculation means 822 outputs an image 

25 similarity, i.e., a similarity of reference images in the 

reference image group, to the retrieval result output means 26. 
Methods of calculating the overall image's similarity include 
finding a total sum of small region similarities for all small 
regions or finding an average value thereof. There is a method 

30 of modifying the overall image's similarity using the match 

determination threshold. The method modifies the similarity to 
1 when the image similarity is equal to or greater than the match 
determination threshold. The method modifies the similarity to 
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0 when the image similarity is smaller than the match 

determination threshold . 

[0148] 

Similar to the second embodiment, the retrieval result 
5 output means 26 outputs a retrieval result. 
[0149] 

The eighth embodiment retrieves images using the image 
similarity calculation method having the effect according to the 
seventh embodiment of the present invention. Differently from 

10 the sixth embodiment, the eighth embodiment calculates a match 
determination threshold for each inquiry image. Accordingly, 
the eighth embodiment can more accurately retrieve all original 
images for edited inquiry images from the reference image group. 
Similar to the seventh embodiment, the eighth embodiment requires 

15 no learning image or device characteristic as input. 
[0150] 

[Ninth Embodiment] 

The ninth embodiment of the present invention will be 
described in further detail with reference to the accompanying 

20 drawings. As shown in a block diagram of FIG. 17, the ninth 
embodiment of the image similarity calculation system according 
to the present invention is functionally composed of the feature 
quantity extraction portion 11, an image similarity calculation 
portion 92, an editing probability model estimation means 93, a 

25 local region weight calculation means 94, and a match 

determination threshold calculation means 95. The feature 
quantity extraction portion 11 is the same as that provided for 
the first embodiment . It may be preferable to configure the local 
region weight calculation means 94 and the match determination 

30 threshold calculation means 95 as single weight value and match 
determination threshold calculation means. 
[0151] 

The editing probability model estimation means 93 
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calculates local region editing probability, i.e., probability 
of an editing process applied to each local region in an image 
from an edited learning image or characteristics of a device to 
perform the editing process supplied as input. The editing 
5 probability model estimation means 93 outputs the calculated 
local region editing probability to the local region weight 
calculation means 94. 
[0152] 

Examples of the device characteristic include editing 
10 patterns applied to an image by the editing device and information 
about the frequency of using each editing patterns. It is 
preferable that the local region corresponds to a small region 
divided by the image division means 111. In addition, the local 
region may correspond to any of multiple smaller regions resulting 
15 from further dividing a small region divided by the image division 
means 111. Moreover, the local region may correspond to each 
element of the image. Furthermore, the local region may 
correspond to any of regions divided in any shapes and sizes 
different from those for the small region division method for the 
20 image division means 111. 
[0153] 

When an edited learning image is input, the local region 
editing probability is measured by specifying regions that an 
editing process was applied in the learning image. Methods of 

25 specifying edited regions may include not only manual methods, 
but also methods of automatically detecting editing processes 
such as the ticker detection method. Characteristics of a device 
to perform editing processes may be supplied as input. In such 
case, there is available the method of measuring the local region 

30 editing probability based on editing patterns applied to an image 
by the device and based on information about the frequency of using 
each editing patterns . 
[0154] 
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The local region weight calculation means 94 calculates a 
local region weight value based on the local region editing 
probability supplied from the editing probability model 
estimation means 93. The local region weight value quantifies 
5 how effectively the feature quantity extracted from local regions 
of an image works for calculation of the similarity for the overall 
image. The local region weight calculation means 94 outputs the 
calculated local region weight value to the image similarity 
calculation means 922 and the match determination threshold 

10 calculation means 95 . It is effective to set a larger local region 
weight value in proportion to a decrease in the local region 
editing probability attributed to the local region. There is an 
example of the method for calculating the local region weight 
value as follows. 

15 Local region weight value = 1 - local region editing 

probability 

The present invention is not limited thereto. During the 
image similarity calculation process, it may be preferable to use 
the other calculation methods capable of eliminating the effect 
20 of a highly possibly edited region. 
[0155] 

Like the local region in the editing probability model 
estimation means 13, it is preferable that the local region 
corresponds to a small region divided by the image division means 

25 111. In addition, the local region may correspond to any of 
multiple smaller regions resulting from further dividing a small 
region divided by the image division means 111. Furthermore, the 
local region may correspond to each element of the image. Still 
further, the local region may correspond to any of regions divided 

30 in any shapes and sizes different from those for the small region 
division method for the image division means 111. 
[0156] 

The match determination threshold calculation means 95 is 

68 



supplied with a local region weight value from the local region 
weight calculation means 94. Based on the local region weight 
value, the match determination threshold calculation means 95 
calculates a match determination threshold for image similarities 
5 found by the image similarity calculation means 922 to be 

described later. The match determination threshold represents 
an optimum threshold value for determining whether or not two 
compared images match. The match determination threshold 
calculation means 95 outputs the calculated match determination 

10 threshold to the image similarity calculation means 922. The 
image similarity calculation means 922 weights the similarity for 
each small region using a small-region-based weight value found 
from the local region weight value. The image similarity 
calculation means 922 calculates the overall image's similarity 

15 by eliminating effects of the editing process. Accordingly, the 
match determination threshold calculation means 95 calculates the 
match determination threshold in consideration for the local 
region weight value . There is a method of calculating the optimum 
threshold value between an edited image and the original image 

2 0 as follows. The match determination threshold calculation means 
95 calculates an estimated value of the overall image ' s similarity 
based on the local region weight value between an edited image 
and the original image by using it and eliminating effects of the 
editing process. Based on the estimated overall image's 

25 similarity, the method finds a threshold value that the 

probability of determining a mismatch (discrepancy) between the 
edited image and the original image is small enough. In addition, 
it may be preferable to provide for effects of feature quantity 
variations due to noise occurrence resulting from re-encoding the 

30 image. 
[0157] 

The image similarity calculation portion 92 functionally 
includes a small region similarity calculation means 921 and an 
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image similarity calculation means 922. 
[0158] 

The small region similarity calculation means 921 compares 
the small region feature quantity of an inquiry image supplied 
5 from the small region feature quantity extraction means 112 with 
that of a reference image supplied as input. The small region 
similarity calculation means 921 calculates a small region 
similarity^ i.e., a feature quantity similarity for each small 
region. The small region similarity calculation means 921 

10 outputs the calculated small region similarity to the image 

similarity calculation means 922. Any method of calculating the 
small region similarity may be settled according to the small 
region feature quantity to be used. For example, when the small 
region feature quantity to be used is a representative value such 

15 as an average value, a mode value, or a median value for the small 
region's color, there is a method of calculating the similarity 
based on a distance value and a correlation value in a color space 
being used. When the small region feature quantity to be used 
is an occurrence histogram for colors and edge elements, there 

20 is a method of calculating the similarity between histograms. 
[0159] 

The small region feature quantity to be used may be Dominant 
Color, Color Layout, Scalable Color, Color Structure, Edge 
Histogram, Homogeneous Texture, Texture Browsing, Region Shape, 
25 Contour Shape, Shape 3D, Parametric Motion, or Motion Activity 
specified in ISO/IEC 15938-3. In this case, similarities can be 
calculated in accordance with the distance (similarity) 
calculation method recommended by ISO/IEC 15938-3. 
[0160] 

30 The similarity may be a binary determination result such 

as 1 or 0. When the edge element feature quantity is used as the 
small region feature quantity, for example, there is a method of 
setting the similarity to 1 if the comparing edge elements match 
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or setting the similarity to 0 if the comparing edge elements do 
not match. It is preferable that a small region concerning the 
small region feature quantity for the reference image supplied 
as input corresponds to a small region concerning the small region 
5 feature quantity for the inquiry image supplied from the small 
region feature quantity extraction means 112. This is because 
corresponding small regions can be directly compared with each 
other. Different image division methods may cause a discrepancy 
between the inquiry image * s small region and the reference image ' s 

10 small region. To solve this problem, it is a good practice to 
use either image's small region feature quantity to complement 
the feature quantity corresponding to the other image ' s small 
region and calculate a similarity. For example, a small region 
concerning the reference image's small region feature quantity 

15 may result from further dividing the small region concerning the 
inquiry image's small region feature quantity into smaller 
multiple regions. In this case, it is possible to generate a 
feature quantity corresponding to the inquiry image's small 
region by calculating an average value of the reference image's 

20 multiple small region feature quantities. 
[0161] 

There may be a case where the small region division method 
concerning the reference image's small region feature quantity 
uses shapes and sizes completely different from those for the 

25 small region division method concerning the inquiry image ' s small 
region feature quantity. In this case, small region feature 
quantities of the reference image's small regions which overlap 
with the inquiry image's small regions are then weighted in 
proportion to overlapping areas and are totaled. In this manner, 

30 it is possible to generate the feature quantity corresponding to 
the inquiry image's small region. 
[0162] 

The small region similarity calculation means 921 inputs 
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the small region similarity to the image similarity calculation 
means 922. The image similarity calculation means 922 weights 
this small region similarity using a weight value for each small 
region found from a local region weight value input from the local 
5 region weight calculation means 94. The image similarity 

calculation means 922 calculates an overall image's similarity 
from the resulting weighted small region similarities . The image 
similarity calculation means 922 modifies the calculated image 
similarity using the match determination threshold supplied from 

10 the match determination threshold calculation means 95 and 

outputs the modified image similarity as an image similarity. 
Methods of calculating the overall image's similarity include 
finding a total sum of weighted small region similarities for all 
small regions or finding an average value thereof. 

15 [0163] 

If the local region of the local region weight value input 
from the local region weight calculation means 9 4 corresponds to 
a small region of the small region similarity input from the small 
region similarity calculation means 921, the local region weight 

20 value can be directly used as a weight value for each small region. 
When the small region of the small region similarity does not 
correspond to the local region of the local region weight value, 
it is a good practice to use the local region weight value to 
complement the weight value corresponding to the small region of 

25 the small region similarity. For example, a local region of the 
local region weight value may result from further dividing a small 
region having the small region similarity into multiple smaller 
regions. In this case, it is possible to generate a weight value 
corresponding to the small region of the small region similarity 

30 by calculating an average value for multiple local region weight 
values . 
[0164] 

There may be a case where the local region division method 
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concerning local region weight values uses shapes and sizes 
completely different from those for the small region division 
method concerning small region similarities. In this case, 
weight values of the multiple local regions which overlap with 
5 the small regions are then weighted in proportion to overlapping 
areas and are totaled. In this manner, it is possible to generate 
the weight value corresponding to the small region of the small 
region similarity. There is a method of modifying the overall 
image' s similarity using the match determination threshold. The 
10 method modifies the similarity to 1 when the image similarity is 
equal to or greater than the match determination threshold. The 
method modifies the similarity to 0 when the image similarity is 
smaller than the match determination threshold. 
[0165] 

15 The ninth embodiment has the effects of the first and fifth 

embodiments. The ninth embodiment can eliminate the effect of 
a highly possibly edited region to calculate the image similarity 
suited for determining whether or not an edited inquiry image is 
generated from the reference image. The ninth embodiment 

20 supplies a local region weight value calculated by the local 
region weight value calculation means 94 to the match 
determination threshold calculation means 95 . According to this 
configuration, the match determination threshold calculation 
means 95 need not recalculate the local region weight value from 

25 the local region editing probability supplied from the editing 
probability model estimation means 93, wherein the local region 
weight value is referenced to calculate a match determination 
threshold. In this manner, the amount of calculation can be 
suppressed. Further, the ninth embodiment needs no process for 

30 detecting an edited region from the inquiry image and calculates 
a local region weight value and a match determination threshold 
from the learning image or the device characteristic. 
Accordingly, another process can be used to find the local region 
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weight value and the match determination threshold in advance, 
making it possible to fast calculate the image similarity. 
[0166] 

The modifications of the first embodiment can be also 

* 

5 applied to the ninth embodiment . That is , the editing probability 
model storage means may be provided instead of the editing 
probability model estimation means 93. It may be preferable to 
provide the editing probability model update means and the editing 
probability model storage means. 

10 [0167] 

[Tenth Embodiment] 

The tenth embodiment of the present invention will be 
described in further detail with reference to the accompanying 
drawings. As shown in a block diagram of FIG. 18, the tenth 

15 embodiment differs from the ninth embodiment as follows. The 
tenth embodiment of the image similarity calculation system 
according to the present invention replaces the image similarity 
calculation portion 92 shown in FIG. 17 according to the ninth 
embodiment with an image similarity calculation portion 1020. 

20 The tenth embodiment includes the reference image group small 
region feature quantity storage portion 25 and the retrieval 
result output means 26. The reference image group small region 
feature quantity storage portion 25 and the retrieval result 
output means 2 6 are the same as those provided for the second 

25 embodiment. 
[0168] 

The image similarity calculation portion 1020 functionally 
includes the small region similarity calculation means 121 and 
the image similarity calculation means 122. 
30 [0169] 

The small region similarity calculation means 121 reads 
small region feature quantities for multiple- reference images 
from the reference image group small region feature quantity 
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storage portion 25 on an image basis. The small region similarity 
calculation means 121 compares the read small region feature 
quantity for each of the reference images with the inquiry image' s 
small region feature quantity input from the small region feature 
5 quantity extraction means 112. The small region similarity 
calculation means 121 calculates a small region similarity, i.e. , 
a feature quantity similarity for each small region. The small 
region similarity calculation means 121 outputs the calculated 
small region similarity for each reference image in the reference 

10 image group to the image similarity calculation means 122. Any 
method of calculating the small region similarity can be settled 
according to the small region feature quantity to be used. For 
example, when the small region feature quantity to be used is a 
representative value such as an average value, a mode value, or 

15 a median value for the small region's color, there is a method 
of calculating the similarity based on a distance value and a 
correlation value in a color space being used. When the small 
region feature quantity to be used is an occurrence histogram for 
colors and edge elements, there is a method of calculating the 

20 similarity between histograms. 
[0170] 

The small region feature quantity to be used may be Dominant 
Color, Color Layout, Scalable Color, Color Structure, Edge 
Histogram, Homogeneous Texture, Texture Browsing, Region Shape, 

25 Contour Shape, Shape 3D, Parametric Motion, or Motion Activity 
specified in international standard ISO/IEC 15938-3. In this 
case, similarities can be calculated in accordance with the 
distance (similarity) calculation method recommended by ISO/IEC 
15938-3. 

30 [0171] 

The similarity may be a binary determination result such 
as 1 or 0 . When the edge element feature quantity is used as the 
small region feature quantity, for example, there is a method of 
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setting the similarity to 1 if the comparing edge elements match 
or setting the similarity to 0 for if the comparing edge elements 
do not match. Different image division methods may cause a 
discrepancy between a small region concerning the small region 
5 feature quantity for the inquiry image input from the small region 
feature quantity extraction means 112 and a small region 
concerning the small region feature quantity for the reference 
image read from the reference image group small region feature 
quantity storage portion 25. To solve this problem, it is a good 

10 practice to use either image's small region feature quantity to 
complement the feature quantity corresponding to the other 
image's small region and calculate a similarity. For example, 
a small region concerning the reference image's small region 
feature quantity may result from further dividing the small region 

15 concerning the inquiry image' s small region feature quantity into 
smaller multiple regions. In this case, it is possible to 
generate a feature quantity corresponding to the inquiry image's 
small region by calculating an average value of the reference 
image's multiple small region feature quantities. 

20 [0172] 

There may be a case where the small region division method 
concerning the reference image's small region feature quantity 
uses shapes and sizes completely different from those for the 
small region division method concerning the inquiry image' s small 

2 5 region feature quantity. In this case, small region feature 
quantities of the reference image's small regions which overlap 
with are then weighted in proportion to overlapping areas and are 
totaled. In this manner, it is possible to generate the feature 
quantity corresponding to the inquiry image's small region. 

30 [0173] 

The image similarity calculation means 122 is supplied with 
the small region similarity for each reference image in the 
reference image group from the small region similarity 
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calculation means 121. The image similarity calculation means 
122 weights the supplied small region similarity using a 
small-region-based weight value found from local region weight 
values supplied from the local region weight calculation means 
5 94. The image similarity calculation means 122 calculates an 
overall image's similarity from the resulting weighted small 
region similarities. The image similarity calculation means 122 
modifies the calculated overall image's similarity using the 
match determination threshold supplied from the match 

10 determination threshold calculation means 95. The image 

similarity calculation means 122 outputs an image similarity, 
i.e. , a similarity of reference images in the modified reference 
image group, to the retrieval result output means 26. Methods 
of calculating the overall image's similarity include finding a 

15 total sum of weighted small region similarities for all small 
regions or finding an average value thereof. 
[0174] 

If the local region of the local region weight value input 
from the local region weight calculation means 94 corresponds to 

20. a small region of the small region similarity input from the small 
region similarity calculation means 121, the local region weight 
value can be directly used as a weight value for each small region. 
When the small region of the small region similarity does not 
correspond to the local region of local region weight value, the 

25 local region weight value may be used to complement the weight 
value corresponding to the small region of the small region 
similarity. For example, the local region of the local region 
weight value may result from further dividing the^ small region 
of the small region similarity into multiple smaller regions. In 

30 this case, averaging multiple local region weight values can 
generate a weight value corresponding to the small region of the 
small region similarity. 
[0175] 
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There may be a case where the local region division method 
concerning local region weight values uses shapes and sizes 
completely different from those for the small region division 
method concerning small region similarities. In this case, 
weight values of the multiple local regions which overlap with 
the small regions are then weighted in proportion to overlapping 
areas and are totaled. In this manner, it is possible to generate 
the weight value corresponding to the small region of the small 
region similarity. There is a method of modifying the overall 
image' s similarity using the match determination threshold. The 
method modifies the similarity to 1 when the image similarity is 
equal to or greater than the match determination threshold. The 
method modifies the similarity to 0 when the image similarity is 
smaller than the match determination threshold. 
[0176] 

Similar to the second embodiment, the retrieval result 
output means 26 outputs a retrieval result. 
[0177] 

The tenth embodiment retrieves images using the image 
similarity calculation method-having the effect according to the 
ninth embodiment. Similar to the ninth embodiment, the tenth 
embodiment requires no process to detect an edited region from 
the inquiry image and makes it possible to previously find the 
local region weight value and the match determination threshold, 
thus enabling fast image retrieval. 
[0178] 

The modifications of the first embodiment can be also 
applied to the tenth embodiment . That is , the editing probability 
model storage means may be provided instead of the editing 
probability model estimation means 93. It may be preferable to 
provide the editing probability model update means and the editing 
probability model storage means. 
[0179] 
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[Eleventh Embodiment] 

The eleventh embodiment of the present invention will be 
described in further detail with reference to the accompanying 
drawings. As shown in a block diagram of FIG. 19, the eleventh 
5 embodiment of the image similarity calculation system according 
to the present invention is functionally composed of the feature 
quantity extraction portion 11, an image similarity calculation 
portion 112 , an editing region detection means 113 , a local region 
weight calculation means 114, and a match determination threshold 
10 calculation means 115. The feature quantity extraction portion 
11 is the same as that provided for the first embodiment. 
[0180] 

The editing region detection means 113 uses a method of 
automatically detecting edited regions such as the ticker 

15 detection method for an edited inquiry image supplied as input. 
Using this method, the editing region detection means 113 
calculates a local region editing probability, i.e., a 
probability that an editing process was applied to each local 
region in the inquiry image . The editing region detection means 

20 113 outputs the calculated local region editing probability to 
the local region weight calculation means 114. The local region 
editing probability may have any probability value or a binary 
value 1 or 0. The local region may correspond to any of multiple 
smaller regions resulting from further dividing a small region 

25 divided by the image division means 111. Moreover, the local 
region may correspond to each element of the image. Furthermore, 
the local region may correspond to any of regions divided in any 
shapes and sizes different from those for the small region 
division method for the image division means 111. 

30 [0181] 

The local region weight calculation means 114 calculates 
a local region weight value based on the local region editing 
probability supplied from the editing region detection means 113 . 
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The local region weight value quantifies how effectively the 
feature quantity extracted from local regions of an image works 
for calculation of the similarity for the overall image. The 
local region weight calculation means 114 outputs the calculated 
5 local region weight value to the image similarity calculation 
means 1122 and the match determination threshold calculation 
means 115. 
[0182] 

It is preferable that the local region corresponds to a small 
10 region divided by the image division means 111. In addition, the 
local region may correspond to any of multiple smaller regions 
resulting from further dividing a small region divided by the 
image division means 111. Moreover, the local region may 
correspond to each element of the image. Furthermore, the local 
15 region may correspond to any of regions divided in any shapes and 
sizes different from those for the small region division method 
for the image division means 111. There is an example of the 
method for calculating the local region weight value as follows . 
Local region weight value = 1 - local region editing 

20 probability 

The present invention is not limited thereto. During the 
image similarity calculation process, it may be preferable to use 
the other calculation methods capable of eliminating the effect 
of a highly possibly edited region. 

25 [0183] 

The editing region detection means 113 may supply local 
region editing probabilities as binary probability values such 
as 1 and 0 . In this case , there is a method of integrating multiple 
local regions into a new local region. The method finds a local 
30 region weight value as a ratio of an area of the local region having 
the local region editing probability value of 0 before the 
integration included in the local region after the integration. 
[0184] 
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The match determination threshold calculation means 115 is 
supplied with a local region weight value from the local region 
weight calculation means 114. Based on the local region weight 
value, the match determination threshold calculation means 115 
5 calculates a match determination threshold for image similarities 
found by the image similarity calculation means 1122 to be 
described later. The match determination threshold represents 
an optimum threshold value for determining whether or not two 
compared images match. The match determination threshold 

10 calculation means 115 outputs the calculated match determination 
threshold to the image similarity calculation means 1122. The 
image similarity calculation means ,1122 weights the similarity 
for each small region using a small-region-based weight value 
found from the local region weight value. The image similarity 

15 calculation means 1122 calculates the overall image's similarity 
by eliminating effects of the editing process. Accordingly, the 
match determination threshold calculation means 115 calculates 
the match determination threshold in consideration for the local 
region weight value. There is a method of calculating the optimum 

20 threshold value between an edited image and the original image 
as follows. The match determination threshold calculation means 
115 calculates an estimated value of the overall image's 
similarity based on the local region weight value between an 
edited image and the original image by using it and eliminating 

25 effects of the editing process. Based on the estimated overall 
image's similarity, the method finds a threshold value that the 
probability of determining a mismatch (discrepancy ) between the 
edited image and the original image is small enough. In addition, 
it may be preferable to provide for effects of feature quantity 

30 variations due to noise occurrence resulting from re-encoding the 
image . 
[0185] 

The image similarity calculation portion 112 functionally 
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includes the small region similarity calculation means 1121 and 

the image similarity calculation means 1122. 

[0186] 

The small region similarity calculation means 1121 compares 
the small region feature quantity of an inquiry image supplied 
from the small region feature quantity extraction means 112 with 
that of a reference image supplied as input. The small region 
similarity calculation means 1121 calculates a small region 
similarity, i.e., a feature quantity similarity for each small 
region. The small region similarity calculation means 1121 
outputs the calculated small region similarity to the image 
similarity calculation means 1122 . Any method of calculating the 
small region similarity may be settled according to the small 
region feature quantity to be used. For example, when the small 
region feature quantity to be used is a representative value such 
as an average value, a mode value, or a median value for the small 
region's color, there is a method of calculating the similarity 
based on a distance value and a correlation value in a color space 
being used. When the small region feature quantity to be used 
is an occurrence histogram for colors and edge elements, there 
is a method of calculating the similarity between histograms . 
[0187] 

The small region feature quantity to be used may be Dominant 
Color, Color Layout, Scalable Color, Color Structure, Edge 
Histogram, Homogeneous Texture, Texture Browsing, Region Shape, 
Contour Shape, Shape 3D, Parametric Motion, or Motion Activity 
specified in ISO/IEC 15938-3. In this case, similarities can be 
calculated in accordance with the distance (similarity) 
calculation method recommended by ISO/IEC 15938-3. 
[0188] 

The similarity may be a binary determination result such 
as 1 or 0. When the edge element feature quantity is used as the 
small region feature quantity, for example, there is a method of 
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setting the similarity to 1 if the comparing edge elements match 
or setting the similarity to 0 if the comparing edge elements do 
not match. It is preferable that a small region concerning the 
small region feature quantity for the reference image supplied 
as input corresponds to a small region concerning the small region 
feature quantity for the inquiry image supplied from the small 
region feature quantity extraction means 112. This is because 
corresponding small regions can be directly compared with each 
other. Different image division methods may cause a discrepancy 
between the inquiry image ' s small region and the reference image ' s 
small region. To solve this problem^ it is a good practice to 
use either image's small region feature quantity to complement 
the feature quantity corresponding to the other image's small 
region and calculate a similarity. For example, a small region 
concerning the reference image's small region feature quantity 
may result from further dividing the small region concerning the 
inquiry image's small region feature quantity into smaller 
multiple regions. In this case, it is possible to generate a 
feature quantity corresponding to the inquiry image's small 
region by calculating an average value of the reference image's 
multiple small region feature quantities. 
[0189] 

There may be a case where the small region division method 
concerning the reference image's small region feature quantity 
uses shapes and sizes completely different from those for the 
small region division method concerning the inquiry image ' s small 
region feature quantity. In this case, small region feature 
quantities of the reference image's small regions which overlap 
with the inquiry image's small regions are then weighted in 
proportion to overlapping areas and are totaled. In this manner, 
it is possible to generate the feature quantity corresponding to 
the inquiry image's small region. 
[0190] 
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The small region similarity calculation means 1121 inputs 
the small region similarity to the image similarity calculation 
means 1122. The image similarity calculation means 1122 weights 
this small region similarity using a weight value for each small 
region found from a local region weight value input from the local 
region weight calculation means 114. The image similarity 
calculation means 1122 calculates an overall image's similarity 
from the resulting weighted small region similarities . The image 
similarity calculation means 1122 modifies the calculated image 
similarity using the match determination threshold supplied from 
the match determination threshold calculation means 115 and 
outputs the modified similarity as an image similarity. Methods 
of calculating the image similarity include finding a total sum 
of weighted small region similarities for all small regions or 
finding an average value thereof. 
[0191] 

If the local region of the local region weight value input 
from the local region weight calculation means 114 corresponds 
to a small region of the small region similarity input from the 
small region similarity calculation means 1121^ the local region 
weight value can be directly used as a weight value for each small 
region. When the small region of the small region similarity does 
not correspond to the local region of the local region weight value, 
it is a good practice to use the local region weight value to 
complement the weight value corresponding to the small region of 
the small region similarity. For example, a local region of the 
local region weight value may result from further dividing a small 
region of the small region similarity into multiple smaller 
regions. In this case, it is possible to generate a weight value 
corresponding to the small region of the small region similarity 
by calculating an average value for multiple local region weight 
values . 
[0192] 
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^ 

There may be a case where the local region division method 
concerning local region weight values uses shapes and sizes 
completely different from those for the small region division 
method concerning small region similarities. In this case, 
5 weight values of the multiple local regions which overlap with 
the small regions are then weighted in proportion to overlapping 
areas and are totaled. In this manner, it is possible to generate 
the weight value corresponding to the small region of the small 
region similarity. There is a method of modifying the overall 

10 image* s similarity using the match determination threshold. The 
method modifies the similarity to 1 when the image similarity is 
equal to or greater than the match determination threshold. The 
method modifies the similarity to 0 when the image similarity is 
smaller than the match determination threshold. 

15 [0193] 

Similar to the ninth embodiment, the eleventh embodiment 
calculates an image similarity suited for determining whether or 
not an edited inquiry image is generated from the reference image 
by eliminating the effect of a highly possibly edited region. In 

20 addition,, the eleventh embodiment calculates a local region 

weight value and a match determination threshold based on the 
local region editing probability directly calculated from the 
inquiry image. The eleventh embodiment calculates a local region 
weight value and a match determination threshold for each inquiry 

25 image differently from the ninth embodiment that heuristically 
finds the local region weight value and the match determination 
threshold as average values from the learning image or the device 
characteristic. Accordingly, the eleventh embodiment can more 
accurately find the image similarity suited for determining 

30 whether or not an edited inquiry image is generated from the 
reference image. The eleventh embodiment requires no learning 
image or device characteristic as input. Similar to the ninth 
embodiment, the eleventh embodiment supplies a local region 
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weight value calculated by the local region weight value 
calculation means 114 to the match determination threshold 
calculation means 115. Accordingly, the match determination 
threshold calculation means 115 need not recalculate the local 
5 region weight value from the local region editing probability 
supplied from the editing probability model estimation means 113 , 
wherein the local region weight value is referenced to calculate 
a match determination threshold. In this manner, the amount of 
calculation can be suppressed. 

10 [0194] 

[ Twel f th Embodiment ] 

The twelfth embodiment of the present invention will be 
described in further detail with reference to the accompanying 
drawings. As shown in a block diagram of FIG. 20, the twelfth 

15 embodiment differs from the eleventh embodiment as follows. The 
twelfth embodiment of the image similarity calculation system 
according to the present invention replaces the image similarity 
calculation portion 112 in FIG. 19 according to the eleventh 
embodiment with an image similarity calculation portion 1220. 

20 The twelfth embodiment provides the reference image group small 
region feature quantity storage portion 25 and the retrieval 
result output means 26. The reference image group small region 
feature quantity storage portion 25 is the same as that provided 
for the second embodiment. 

25 [0195] 

The image similarity calculation portion 1220 functionally 
includes a small region similarity calculation means 1221 and an 
image similarity calculation means 1222. 
[0196] 

30 The small region similarity calculation means 1221 reads 

small region feature quantities for multiple reference images 
from the reference image group small region feature quantity 
storage portion 25 on an image basis . The small region similarity 



calculation means 1221 compares the read small region feature 
quantity for each of the reference images with the inquiry image* s 
small region feature quantity input from the small region feature 
quantity extraction means 112. The small region similarity 
5 calculation means 1221 calculates a small region similarity, i.e. , 
a feature quantity similarity for each small region. The small 
region similarity calculation means 1221 outputs the calculated 
small region similarity for each reference image in the reference 
image group to the image similarity calculation means 1222. Any 

10 method of calculating the small region similarity can be settled 
according to the small region feature quantity to be used. For 
example, when the small region feature quantity to be used is a 
representative value such as an average value, a mode value, or 
a median value for the small region's color, there is a method 

15 of calculating the similarity based on a distance value and a 
.correlation value in a color space being used. When the small 
region feature quantity to be used is an occurrence histogram for 
colors and edge elements, there is a method of calculating the 
similarity between histograms. 

20 [0197] 

The small region feature quantity to be used may be Dominant 
Color, Color Layout, Scalable Color, Color Structure, Edge 
Histogram, Homogeneous Texture, Texture Browsing, Region Shape, 
Contour Shape, Shape 3D, Parametric Motion, or Motion Activity 
25 specified in international standard ISO/IEC 15938-3. In this 
case, similarities can be calculated in accordance with the 
distance (similarity) calculation method recommended by ISO/IEC 
15938-3. 
[0198] 

30 The similarity may be a binary determination result such 

as 1 or 0 . When the edge element feature quantity is used as the 
small region feature quantity, for example, there is a method of 
setting the similarity to 1 if the comparing edge elements match 
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or setting the similarity to 0 if the comparing edge elements do 
not match. Different image division methods may cause a 
discrepancy between a small region concerning the small region 
feature quantity for the inquiry image input from the small region 
5 feature quantity extraction means 112 and a small region 

concerning the small region feature quantity for the reference 
image read from the reference image group small region feature 
quantity storage portion 25. To solve this problem, it is a good 
practice to use either image's small region feature quantity to 

10 complement the feature quantity corresponding to the other 

imagers small region and calculate a similarity. For example, 
a small region concerning the reference image's small region 
feature quantity may result from further dividing the small region 
concerning the inquiry image* s small region feature quantity into 

15 smaller multiple regions. In this case, it is. possible to 

generate a feature quantity corresponding to the inquiry image's 
small region by calculating an average value of the reference 
image's multiple small region feature quantities. There may be 
a case where the small region division method concerning the 

20 reference image's small region feature quantity uses shapes and 
sizes completely different from those for the small region 
division method concerning the inquiry image's small region 
feature quantity. In this case, small region feature quantities 
of the reference image's small regions which overlap with the 

25 inquiry image's small regions are then weighted in proportion to 
overlapping areas and are totaled. In this manner, it is possible 
to generate the feature quantity corresponding to the inquiry 
image ' s small region . 
[0199] 

30 The image similarity calculation means 1222 is supplied 

with the small region similarity for each reference image in the 
reference image group from the small region similarity 
calculation means 1221. The image similarity calculation means 
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1222 weights the supplied small region similarity using a 
small-region-based weight value found from local region weight 
values supplied from the local region weight calculation means 
114. The image similarity calculation means 1222 calculates an 
overall image's similarity from the resulting weighted small 
region similarities. The image similarity calculation means 
1222 modifies the calculated overall image' s similarity using the 
match determination threshold supplied from the match 
determination threshold calculation means 115. The image 
similarity calculation means 1222 outputs an image similarity, 
i.e. , a similarity of reference images in the modified reference 
image group, to the retrieval result output means 26. Methods 
of calculating the overall image's similarity include finding a 
total sum of weighted small region similarities for all small 
regions or finding an average value thereof. 
[0200] 

If the local region of local region weight value input from 
the local region weight calculation means 114 corresponds to a 
small region of the small region similarity input from the small 
region similarity calculation means 1221, the local region weight 
value can be directly used as a weight value for each small region. 
When the small region of the small region similarity does not 
correspond to the local region of local region weight value, the 
local region weight value may be used to complement the weight 
value corresponding to the small region of the small region 
similarity. For example, the local region of the local region 
weight value may result from further dividing the small region 
of the small region similarity into multiple smaller regions. In 
this case, averaging multiple local region weight values can 
generate a weight value corresponding to the small region of the 
small region similarity. 
[0201] 

There may be a case where the local region division method 
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concerning local region weight values uses shapes and sizes 
completely different from those for the small region division 
method concerning small region similarities. In this case, 
weight values of the multiple local regions which overlap with 
5 the small regions are then weighted in proportion to overlapping 
areas and are totaled. In this manner^ it is possible to generate 
the weight value corresponding to the small region of the small 
region similarity. There is a method of modifying the overall 
image* s similarity using the match determination threshold. The 
10 method modifies the similarity to 1 when the image similarity is 
equal to or greater than the match determination threshold. The 
method modifies the similarity to 0 when the image similarity is 
smaller than the match determination threshold. 
[0202] 

15 Similar to the second embodiment, the retrieval result 

output means 2 6 outputs a retrieval result. 
[0203] 

The twelfth embodiment retrieves images using the image 
similarity calculation method having the effect according to the 

20 eleventh embodiment. Different from the first embodiment, the 
twelfth embodiment calculates a local region weight value and a 
match determination threshold for each inquiry image. 
Accordingly, the twelfth embodiment can more accurately retrieve 
all original images for edited inquiry images from the reference 

25 image group. Similar to the eleventh embodiment, the twelfth 
embodiment requires no learning image or device characteristic 
as input . 

Industrial Applicability 
30 [0204] 

The present invention is applicable to the use of an image 
similarity calculation system, an image similarity calculation 
method, an image similarity calculation program for calculating 
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similarities of multiple images, an image retrieval system for 
retrieving an image similar to an inquiry image, and the like. 
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